Using LIME with Vision Transformers (VIT)

LIME (Local Interpretable Model-agnostic Explanations) has always been billed as the "model agnostic" way to apply explainable AI (XAI) to any sort of ML model. However, if you have tried to use it with Vision Transformers, it essentially fails.  In fact, several sources flat-out say it can't be done.

I've found a way to utilize LIME with VITs and will be sharing it in this post.

Along with text explainability libraries, LIME provides a set of functions for image explainability as well.  To work with LIME in your vision project, you will need to add these modules:

from lime import lime_image

from skimage.segmentation import mark_boundaries

Later, after your model is setup and ready, use lime_image like this:

explainer = lime_image.LimeImageExplainer()

Next, with the explainer object call explain_instance, which takes in an image or list of images and a predict function.  Here we are tempted to put in the model.predict function for our Vision Transformer.  That is wrong.  In fact, you must make a helper function that manipulates the data before calling in for the prediction:

explanation = explainer.explain_instance(image_list.astype('double'), 

pred_fn, 

top_labels=3, hide_color=0, num_samples=1000)

The first parameter is a numpy array of the image(s) I want analyzed.  *Important* - it does not contain an extra batch dimension usually already added.  That's added later in the helper function.

The second parameter is the name of the helper function called prod_fn.  This function allows me to add the extra dimension needed by a Vision Transformer.  Make sure when you call the explain_instance, that the input images in the first parameter do not have that first dimension yet.

Here is my implementation of the pred_fn which will take a single image or a list of them.

def pred_fn(imgs):

tot_probs = []

for img in imgs:

# Add the explanation dimension

exp_img = np.expand_dims(img, axis=0)

# Make the prediction

img_pred, _ = vit_model.predict(exp_img)

# Add the predictions to a list to be returned to LIME

tot_probs.append(img_pred[0])

return tot_probs

The output of the predict function is a 2D list.  We only want the first dimension to be added to the return list.

Finally, the explanation can be used to mask off areas of the original image most salient to the model's output.

temp_1, mask_1 = explanation.get_image_and_mask(

explanation.top_labels[0], positive_only=True, 

num_features=3, hide_rest=True)

temp_2, mask_2 = explanation.get_image_and_mask(

explanation.top_labels[0], positive_only=False, 

num_features=3, hide_rest=False)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15,15))

ax1.imshow(mark_boundaries(temp_1, mask_1))

ax2.imshow(mark_boundaries(temp_2, mask_2))

ax1.axis('off')

ax2.axis('off')

ax1.set_title('')

ax2.set_title('')

fig.tight_layout()

The process outputs these beautiful image explanations:

Original image - The model correctly labeled the image, "golden_retriever"

The first masked image that shows those areas most salient to the decision.  Note that the black background is due to the parameter "hide_rest" being set to True. 

The second masked image again shows the areas most salient to the decision but this time in context with the rest of the image.

I hope this is helpful in your Vision Transformer explanations!

Multiline Javascript with PUG Templates - Mismatched Bracket Error

I came across a straight-forward problem with no apparent solution.  Once I finally fixed it, I wanted to show everyone the solution I found.  Maybe there is a better way?

If you are writing pug temples with multiline javascript, you might get this error:

SyntaxError: Mismatched Bracket: }

That's unfriendly!

(10 minutes later)... I finally figured out the fix was to add a period after the script tag, like this:

script(type="text/javascript").

var recorder = document.getElementById("recorder");

var player = document.getElementById("player");

(See the period after the script close paren?)

Absolute Awesome Terminal Setup and Customization

A couple of my students were asking about my terminal and what I do to customize it.  I'll say, it is awesome!

To those who don't spend a lot of time in a terminal, it may sound stupid to highly customize it.  However, when it's where you spend a lot of your time programming, it's a big help to get it just right.

To start out, I recommend "The Pug Engineer's" guide to setting up your term: https://github.com/pugengineer/iTerm2-ohmyzsh-powerlevel9k

I sometimes use iTerm for my terminal as Pug recommends.  However, because I use VS Code so much, I usually just use the terminal in VS Code.  So, I don't get some of the advantages of iTerm such as Profiling, Hot Keys, etc.  VS Code's terminal can use zsh, autocomplete, etc, so many of the advantages are there.

And I do vary my settings from Pug.  Here are the changes:

• I use the "agnosterzak" theme for zsh.  You can find it along with setup instructions here: https://github.com/zakaziko99/agnosterzak-ohmyzsh-theme.  I found the Powerline9 theme to be too long.  Even Agnosterzak takes up too much space.  So, I modified it to just give time, git status, and a prompt sign.  I got rid of the username and directory listing.  They just made the status line too big.
  
  
• In VS Code I like to use a Powerline font (https://github.com/powerline/fonts).  I currently enjoy "Inconsolata for Powerline" as it's non-serif and easy on the eyes for me.  To add a Powerline font in VS Code, select Code => Preferences => Settings, then search for "Editor: Font Family".  Add your favorite Powerline font to the start of this box (comma delimited).
  
  
• For VS Code, I use the Dark+ theme.

That's it.  It's not too much work to set up... maybe a Saturday morning's worth of time.  However, it will pay big dividends on your next programming project.

Getting Rid of Console Window in OpenFrameWorks for Windows

Yes, this is one of the most aggravating things about OpenFrameworks within Visual Studio. When you run the project, it always pops open a console window that needs to be closed separately.

So, the first thing I do is run the procedure below to correct this problem.  Quick and easy, but a little annoying.  Let me know if this works for you.

Scripting Jobs in SQL Server

Have you every wanted to save your SQL Server jobs to a SQL script file?  It's not straight-forward from the interface, but is easily done.  Just follow these steps:

1. In SQL Enterprise Manager, open the list of Jobs by opening the Object Explorer tree to SQL Server Agent -> Jobs
2. Next, press F7 to open the "Object Explorer Details" (Can also use the menu View -> Object Explorer Details)
3. You should see all your jobs listed.  If not,  you might need to navigate through the list and find them.
4. Select the jobs you want to backup (You can use Shift and Ctrl)
5. Right click and select Script Job As -> Create To -> File
6. Finally enter a filename and press Save

I hope this makes your life easier.  Now get to backing up those jobs!

Macbook Pro GPU Problem Fix

This can't be good!

If your Macbook is giving you a white screen of death during boot up, you might want to try this procedure.  My 2012 Macbook was getting about 1/3 of the way through the boot process, blinks the screen, coughs, then starts over.

It turns out a major problem with older macs is the AMD GPU eats itself and prevents bootup.  You have two options: go to the Apple Store and pay big dollars for a new graphics card, or disable the GPU.

If you decide to disable the GPU, just know that 3D applications that normally use the GPU will be slow.  But for normal operations, your Mac will work fine.

Here is the procedure I successfully used:

Reset the SMC

Shutdown, unplug everything except power and hold

leftShift + Ctrl + Option + Power 

Hold this for 5 seconds

Reset NVRAM: Power up and hold

Command + Option + p + r

Until you hear the startup chime two times.

Shutdown computer

Power up and boot into Single User Recovery by holding

if you are on high sierra 10.13.6+ you might need to use Command + r instead

Command + r + s

On the screen that loads choose a language. Next, on the screen which appears next choose the following options in the menu Utilities -> Terminal (see below.)

In the terminal that comes up, type the following commands:

Disable SIP (This is slow)

csrutil disable

Disable Discrete GPU on boot by running

nvram fa4ce28d-b62f-4c99-9cc3-6815686e30f9:gpu-power-prefs=%01

Reboot

reboot

Boot into Single User-mode by holding

Command + s

It might look as if it hanged, but press enter and you should see the shell (root#)

Mount root partition writeable

/sbin/mount -uw /

Make a kext-backup directory

mkdir -p /System/Library/Extensions-off

Move ONLY ONE offending kext out of the way

mv /System/Library/Extensions/AMDRadeonX3000.kext  /System/Library/Extensions-off/

Inform the system to update its kextcache:

touch /System/Library/Extensions/

Reboot

reboot

I hope this helps you get more life out of your Macbook.  Let me know if this worked.

Creating JSON in C# With Dynamic

I'm excited to share a new (to me) way of creating JSON on-the-fly.  It uses the dynamic type in C#, which I knew about, but hadn't used too much.  Now I see how powerful it really is.

I ran across a problem where I had to build a JSON string to send to a web service as a parameter.  It was short but nested several layers deep.  I also needed it to be highly configurable.  I could build it with a string, or better yet, a StringBuffer.  Or I could build an object, then serialize it to JSON.  The former seems fragile, the later feels like overkill (And, there is nothing worse than trying to decode what another programmer has done with a thousand little classes you have to hunt down.)

Enter the dynamic type.  I knew it was very late-binding, thus avoiding compile-time type checking.  It gets its type at run-time.  So, it infers the type when the program runs and is able to do this:

dynamic nVal = 23.4;
dynamic sVal = "Hello";
Console.WriteLine(nVal.GetType().ToString());   // System.Double
Console.WriteLine(sVal.GetType().ToString());   // System.String

However, a little talked about feature is that it is an object that can be a container too.  And it can contain different types including arrays and objects.  This sets us up to be able to nest JSON very nicely.  Check out this example:

dynamic message = new JObject();
product.to = "UserToken";
product.priority = 5;
product.notification = new JObject();
product.notification.message = "You have a new order.";
product.notification.sound = "default";

Console.WriteLine(message.ToString());

// Output JSON
{
    "to": "UserToken",
    "priority": 5,
    "notification": {
        "message": "You have a new order.",
        "sound": "default"
    }
}

Hopefully, this will help you create simpler JSON data too.