I have previously written about the SVM Kernel Trick and described how the polynomial kernal allows for massive programmatic feature expansion without massive increase in compute for inner product distance calculations. However, there is another, even more powerful kernel available for SVM — the Radial Basis Function or RBF.

A couple of big differences:

1. The RBF kernel uses Euclidean distance (typically) rather than inner product for distance calculations
2. The RBF kernel generates a similarity score (a percentage value between 0% and 100%)
3. The RBF kernel does not explicitly add features to data, but implicitly is equivalent to having infinite dimensions (which makes it an extremely powerful classifier)
4. Despite this “infinite dimensionality” the RBF kernel is able to make really fast inference — making it both faster *and* more powerful than the polynomial kernel

Points 3 and 4 are especially bold claims. How can this be both infinitely flexible *and* highly performant? Let’s start with the math.

Generating Similarity Scores

Similarity of test point to data =
Exponent of (negative (euclidean distance squared) divided by (sigma squared))

https://colab.research.google.com/drive/1enYEfZJLYULQ-7qhgdoRQrCBAkfArl7Y#scrollTo=lnNWFBeFqrLv

When two points are identical (distance = 0) you get a similarity score of 1 (or 100%). As the distance grows, the similarity drops. The value for sigma determines the upper distance boundary before distances get evaluated to 0 — in Colab, with a sigma value of 1, you hit that boundary around 28 or so, however if you raise sigma, you are able to continue to differentiate between higher distance values (if needed).

So we can calculate a distance between training data and test points and simply pick the most similar point as our prediction. Infinite flexibility!

Avoiding High Compute Costs at Inference

But wait. We could do this with k-NN as well, and it came with a serious drawback — at inference, the calculation of distance between a test point and every training point is extremely computationally expensive. Wouldn’t we have the same problem with SVM?

Here’s where the genius part comes in: Once your SVM model is built, you don’t actually need to keep all of your training data and compare the test points to every training point. You just need the support vectors.

In case you’ve forgotten, support vectors are the training data points that are used to determine the placement of the boundary for SVM. They are equally distant from the decision boundary.

Thus, when you are comparing similarities, if you simply look at the similarity scores of your test point against the support vectors in your training data, whichever is most similar = your prediction! You only need a relatively small number of distance calculations after the model has been built to do inference, making this *extremely* fast.

Summary

In summary, the RBF kernel for SVM is an incredibly powerful and performant modeling approach when facing extremely complex data. It has the same flexibility as k-NN, however at inference distances only need to be calculated from a small number of training data points, making it highly performant as well.

For a simple Naive Bayes classifier, we deal with two types of distributions for data and predictions: binomial and normal.

Binomial Distributions: Pick a Side

A binomial distribution is one in which all possible values are either 0 or 1. Another way to think about this is “inactive” vs. “active,” 0% vs. 100%, or “False” vs. “True.” In any case, binomial distributions in data have only two possible values of 0 and 1.

This becomes important when building certain types of feature vectors / embeddings. You might be taking unstructured data (such as letters or words) and converting it into a feature vector in order to perform math on it. Each feature or column in your data represents the presence or absence of a certain idea — for example, “the first character is a vowel.” This is either true or not, so the value is either going to be 1 for true or 0 for false. There is no sense in saying “kinda” — at least not in the English language — and so there is no logical reason for a value to be anything other than 1 or 0. The feature vector in this case is a collection of these columns where whatever you are evaluating is being checked for truth vs. falsehood, or active vs. inactive, and coded as a 1 or 0 accordingly.

For example: say I define a vector / embedding scheme that has four features: first character vowel, last character vowel, more than five characters in word, and more vowels than consonants. If I take the word “language” and use this logic, I would convert it into this four-dimensional feature vector as follows: 0 1 1 0

Normal Distributions: Freedom and Flexibility

In contrast to binomial distributions, normal distributions are those where the features can be any value from 0 to 1. We make assumptions about how features are distributed via the famous bell curve (meaning mean, median, and mode values are all going to be the same), but the actual value of a given feature can be literally any value between 0% and 100%.

Combining the Distributions for Naive Bayes

We end up using both of these concepts in Naive Bayes. We have training data that contains multiple observations of something that we convert into binomial feature vectors, and then we separate them by class and create normal probability distribution vectors which we can use to make predictions.

Using Distributions to Make Predictions with Naive Bayes

Say for example we have four keywords in a document that is classified as “academic” for our purposes: language, speak, analog, and alibaba. This creates a data matrix for our academic class of:

0 1 1 0
0 0 0 0
1 0 1 0
1 1 1 1

Because we are assuming a normal distribution, we can calculate the average values for each of our features to come up with a probability vector which we can use for Naive Bayes predictions:

.5 .5 .75 .25

If I have a new word — clown — and I need to determine the probability that it belongs in the academic category, I can use the chain rule and Bayes Theorem to calculate the probability that this word belongs in that class. (I won’t go into it here, but the answer is about 4.7%.)

In order to check if this is the right prediction to make, I need another class to compare it to. Let’s assume I had other words that fit into a category of “fun” and after creating their embeddings and calculating the average feature values, they had the following probability vector:

.2 .3 .4 .6

(Obviously this training data had a different number of observations than our first class. There’s no rule against that!)

We can now again use the chain rule and Bayes Theorem to come up with an aggregate probability. (This time it’s around 13.4%.)

Now I can make my prediction — since 13.4% probability is more likely than 4.7% probability, if I am choosing between those two classes, I predict that clown belongs in the “fun” class. (With apologies to any of my friends who took a challenging academic path in clown school…)

Practical Application

This simple example actually calls out a number of core tenets of data science. If you know what signals are valuable in a certain type of data (my four-feature example probably wouldn’t be great in real life, but might not be a bad start for a higher dimensional vector / embedding scheme…) then you can create the embeddings using rules that extract those meanings via binomial distributions. You can then take two or more classes of data, convert each relevant observation into an embedding, and then take the average values for each feature in each class to come up with a normal distribution probability vector. This probability vector, plus some basic statistics and the naive assumption that allows us to use the chain rule to calculate aggregate probabilities, can then be applied to an unknown observation and used to find the most likely class that this new observation belongs to, which we can then use as our prediction in a machine learning model.

You will see this basic principle applied all over machine learning today, well beyond the simple Naive Bayes example illustrated here, and understanding how this works will help you wrap your arms around much more complex topics in data science.

L1 Definition

L1 regularization looks very similar to L2, but it is doing something dramatically different. Whereas L2 regularization penalty was the sum of the squared values of the model weights times a constant, L1 regularization is the sum of the absolute values of the model weights times a constant.

L1 Loss = ((Sum of Absolute Weights) * A Constant) + Model Loss

The goal of L1 regularization is to minimize the overall total weight values (thus minimizing the L1 loss penalty), and if possible, reducing as many of them as possible all the way to 0 or close to it while continuing to minimize model loss. Whereas L2 regularization sought to smooth out values between weights, and as such actually amplified features that would otherwise have smaller weights, L1 regularization will put as much weight as possible on as few features as possible, allowing us to completely ignore certain features altogether.

Similar to the previous blog example, say you have a model that has an overall loss of 56 without regularization. This model has three features, and the weights for each feature are as follows: w0 = 10, w1 = -4, and w2 = 1.5.

What does this look like with L1 regularization? For now, we’ll assume our hyperparameter constant value is set to 1. Thus, the imputed loss of our model under L1 regularization equals:

((Sum of Absolute Weights) * A Constant) + Model Loss

((10 + 4 + 1.5) * 1) + 56

…or 71.5. Again, the higher loss value is an arbitrary value — this is the same model that without regularization had a loss value of 56. Nothing has changed because we’re adding the regularization term.

However, here’s where things get interesting. With L1 regularization, we have a simple question to ask: if I decrease the magnitude of a feature weight by x, is the additional Model Loss greater than or smaller than x? If the answer is that the Model Loss increase is less than the weight value, then the algorithm will make the model weight smaller until this is no longer true.

Let’s take a look at a couple of additional models and their unregulated loss value:

Without regularization, Model 1 would have been selected as the best of the three by our algorithm. However, now if we apply our L1 regularization with a constant value of 1:

The other two models are now judged to outperform Model 1, and Model 2 gives us the best overall L1 regularized loss. Note that in this version of the model, we no longer need to consider w2 — the weight is zero, so for future versions of the model we build out, we can drop this feature altogether.

As with L2 regularization, we can weight the regularization effect more heavily and see if and how that changes our evaluation. Say for example we set the constant to 3 instead of 1:

The more weight you give to the L1 regularization term, the better Model 3 is going to look since it has the lowest overall sum of absolute model weights, and indeed at any C value higher than 2, Model 3 is going to be evaluated as the lowest loss model. Even if we set the C value to 2 and the L1 regularized loss of Models 2 and 3 were equal, Model 3 would still be considered the superior model because it allows us to ignore both w1 and w2.

We could generate a final version of the model that only contained w0 and base all of our test predictions on that one feature. We no longer need to consider w1 and w2, as long as the L1 regularization assumptions being made were accurate to begin with.

Practically speaking, you would almost never use L1 regularization on a model with only three features. In small data situations like this, you’re probably looking for data addition and enrichment opportunities rather than feature rationalization. However, let’s say your data contained three-thousand features instead of just three. Depending on your modeling approach, this can take model training and serving time up fairly dramatically, potentially to unworkable levels. In this situation, creating a baseline model and using L1 regularization to identify features that can be dropped to 0 or close to it would be extraordinarily valuable, as the compute time to model around 300–400 features can be orders of magnitude less than 3,000 or more. Even if you give up a little in the way of predictive performance, you’ve gained a lot in the flexibility and clock-time performance of the models on smaller datasets.

Elastic Net

Elastic Net combines L1 and L2 regularization. It computes both values using the same procedures as before and then weights each one according to the relative weight you want to give L1 vs. L2 regularization. (The two weights will equal 100%.) You could write it out like this:

(L1 percent) * ((Sum of Absolute Weights) * the Constant) +
(1 — L1 percent) * ((Sum of Squared Weights) * the Constant) +
Model Loss

So if you wanted to try building a model with a 50/50 split between L1 and L2 regularization, you’d set the L1 percent value to 0.5. If you wanted to do mostly L1 regularization but throw in just a little L2 regularization effect, you might set the L1 percent value to 0.9. And so on.

In the end, the best values and weights might be guessable based on your knowledge of the dataset, but you might also need to try various combinations via grid search or some derivative of it in order to find the optimal values.

Summary

In summary — whereas L2 regularization tends to smooth out the magnitude of the features in your dataset, L1 regularization goes the opposite direction and tries to minimize the number of features that have to be considered at all, even if it means increasing the magnitude of already large variable weights. The right choice between the two depends on the nature of the data you are working with. In situations where both might be useful to more or less degree, you can use Elastic Net and adjust the relative weighting of the regularization penalties until you find the optimal solution. Regardless of the regularization approach, your end goal is to produce models that perform as well as possible in the real world. For L2 regularization, this usually means better overall predictions. While this might be true for L1 regularization as well, the focus is usually on eliminating extraneous features from the data when you have too many of them for the compute resources you have available, thus allowing you to build and iterate future models on smaller subsets of data.

As with everything in data science, a great deal of hypothesis building and experimenation is required in order to come up with the optimal values and approach for a given data problem, and in some cases you may find your data reflects reality well enough that you don’t need to use regularization at all! The larger and more representative your data are, the less benefit regularization will provide. The only way to know for sure in most cases is trial and error.

One of those core conceptual misunderstandings is L2 regularization, and so I’m going to dissect that concept in simple terms here.

Also, I’m going to avoid any jokes pertaining to “regularity” in any other sense of the word. You’re welcome.

Why Consider L2 Regularization?

L2 regularization begins with an assumption that extremes exist in our data that are not reflective of reality. It is a mathematical tool to penalize a model for making extreme judgements about a feature. The goal is to compensate for idiosyncrasies in data that can lead to models that make suboptimal decisions in the real world.

Every feature (or variable) in a dataset is subject to potential relative sampling error. By “relative” I mean a feature value risks either being generally more impactful or generally less impactful than that feature actually is in the real world. This presents problems when building parametric machine learning models because they’re going to come to an optimal solution based on the data they have available, and it’s not until you try making predictions using that solution that you discover it doesn’t line up as well as you’d hope to reality.

L2 regularization results in a model with an overall increase in the loss value on the training data used to build the model. If our data is actually reflective of the real world, this would be a bad thing. However if the assumption about extremes is correct, the adjusted model will perform better on real-world, unseen data, which is the ultimate goal of all machine learning.

L2 Regularization Definition and Considerations

L2 regularization adds a penalty term to your model’s loss value based on squaring the feature weights the model computes, summing them all up, and multiplying the result by a tunable hyperparameter (usually referred to as either “alpha” or “lambda” depending on the author.) I will refer to this value as simply a constant.

L2 Loss = ((Sum of Squared Weights) * A Constant) + Model Loss

For example, say you have a model that has an overall loss of 32 without regularization. This model has three features, and the weights for each feature are as follows: w0 = +4, w1 = +0.1, and w2 = -1. For now, we’ll assume our hyperparameter constant value is set to 1. Thus, the imputed loss of our model under L2 regularization equals:

((Sum of Squared Weights) * A Constant) + Model Loss

((16 + 0.01 + 1) * 1) + 32

…or approximately 49. (The effect of the 0.01 on the model is small enough we can ignore it for practical purposes.)

Note that this higher loss value is just an arbitrary number. We have not changed the model weights at all, so the L2 regularized value of 49 is equivalent to the non-regularized value of 32 because it is literally the exact same predictive model.

We’ll look at some additional examples in the next major section and how this modifies the model in the end. For now, as long as you understand how we get to the value of 49 here, you’re good to go.

Parametric Models Only

Because L2 regularization is applying a penalty based on feature weights, only modeling approaches that make decisions based on weighting features are candidates for L2 regularization. Thus, non-parametric modeling approaches such as nearest neighbors or decision tree-based models are not candidates for L2 regularization. Non-parametric models have other ways of accomplishing similar things (for example, increasing k for kNN or lowering the max depth for decision trees) but technically speaking, they do not employ L2 regularization in the mathematical sense.

How Much Do You Trust Your Data?

The overall impact of L2 regularization is tuned via the constant hyperparameter. If you believe your data are generally pretty representative of reality and you just want to nudge model weights a bit, you can set this hyperparameter value to something small (say, 0.1 or 0.01 perhaps) and the overall effect of L2 regularization will be present but minimized. On the other hand, if you believe your data are likely not representative of reality (perhaps you simply don’t have enough of them, or maybe the data collection practices leave something to be desired) then you can set the hyperparameter value to something large (maybe a value between 2 and 10 or larger, depending on your data…) so that it will have a substantial impact on the overall model weights.

For example, in our first calculation where we reached a loss value of 49 when the constant value was 1, setting the constant to 0.1 would have resulted in:

((16 + 0.01 + 1) * 0.1) + 32

Again, we can ignore the 0.01, so we end up with 17 * 0.1 + 32 or 33.7 as our imputed loss. The L2 regularization didn’t have much effect on the loss value at all.

On the other hand, if we had used a constant value of 10, the imputed loss value would have been 170 + 32, or 202. In this scenario, the L2 regularization has a bigger impact on the loss value than the model loss itself.

Again, we’ll look at the ramifications here in a bit. For now, just make sure you understand how adjusting the constant value changes the overall loss value calculations.

As a reminder, in the examples we’ve used so far, the model weights themselves have not changed. All we’ve done so far is look at different ways to impute a loss value on the exact same model.

The End Result

In order to get to the optimal solution, L2 regularization will tend to reduce the magnitude of large absolute weight values and conversely may increase the magnitude of smaller absolute weight values. Put another way, all else equal it will tend to try to make weights more similar to each other (magnitude-wise) than they might otherwise be without L2 regularization. It’s a smoothing function.

Examples

Machine learning algorithms either use matrix multiplication or gradient descent (or combinations and derivatives of these) to determine the set of weights that produce the lowest possible loss value. Let’s examine three possible values along the way for our model that might have been intermediate points along the way in finding the optimal example:

Without L2 regularization, the model simply finds the values that produce the lowest model loss, so in this case Model 1 would be deemed the “optimal” model based on our training data.

Let’s say we suspect we might benefit from L2 regularization, so we decide to apply it with a Constant (lambda/alpha) value of 1. Now when we look at the loss values:

…notice that it is now Model 3 that gives us the lowest loss value, not Model 1 any more. Thus, the regularized best model has changed based on our decisions and approach.

Note that in this example compared to Model 1 we have reduced w0 by 25%, but we have increased w1 by 1,000%, and increased w2 by 100%. The absolute value sum of our weights is actually larger for Model 3 (6) than it is for Model 1 (5). (This will come back if we look at L1 regularization in a future post…)

We made much larger percentage moves in w1 and w2, but because they started at smaller magnitude scales, the overall change based on L2 regularization was a lower loss (14 vs. 17) which overcame the difference in the original Model Loss (34 vs. 32).

Changing the Constant

What does this look like if we decide to give more weight to the regularization effect? Let’s change the C constant to 2:

Note that now Model 2 would be deemed to be the best model of the three.

If we were to make the L2 regularization constant a small value, say 0.1 or 0.2, then the effect wouldn’t be big enough to change the overall modeling approach and Model 1 would be picked as the optimal model.

Once again, let me reiterate — The increasing or decreasing loss values due to regularization do not change the model weights themselves. All that we are doing is determining which model is best based on the combination of the model loss plus the L2 regularization effect.

How do we know what the best value for our constant C is? You’d need to test it just like anything else, and perhaps employ Grid Search or some variation in order to find the optimal value that gave you the best possible validation loss. Training loss is only relevant as the model loss input to the overall loss calculation when picking the best model to validate.

Summary

Hopefully this blog post helps make clear the purpose of L2 regularization, how it works, and the impact it has based on the constant (alpha/lambda) hyperparameter value applied.

This fall I am teaching a Data Management course as part of Saint Louis University’s M.S. in Health Data Science program. Over the summer, two other professors and I had a brainstorming session where we tried to think through what seemed like a hard problem in academia — the proliferation of LLMs like ChatGPT and others that can take relatively simple prompts and turn them into code and explanations of results, and even full paper write-ups that typically made up assignments for a number of classes.

Plagiarism in academia has been a longstanding issue. Search engine results, Stack Overflow posts, and even sites listed as “homework aids” have been providing code solutions that get copied and pasted into assignments for a very long time. Copying and pasting lightly modified search results like papers or articles as original work is nothing new. But tools like Duet (shown above), ChatGPT, and Bard just make it easier to produce higher quality, seemingly original work with very little effort.

However, this is only in academia where this is perceived as a problem. In the corporate world, if an LLM could reduce the amount of work to finish a project by 90%, we’d celebrate it as a productivity tool. So the question my fellow professors and I asked ourselves was, “Is there a way to embrace LLMs and search results while still upholding academic integrity and ensuring actual learning is taking place?”

This is not unlike the advent of the cheap, widely available calculator in math education. If you’re a high school math teacher and you send students home with 100 simple math problems as homework, how would you know if they used a calculator to do them? You probably wouldn’t. So is math education ruined? Of course not! However, you do have to adapt. You simply assume every student has access to a calculator (and today, probably a graphing calculator that has complex functions built in.) And if that’s the case — and here’s the beautiful part of the brave new world — you simply get to ask harder, and I would argue, more meaningful questions.

Calculators did not ruin math education. They made it infinitely better. They allow us to pose challenges that force students to display conceptual understanding as opposed to computational prowess. While there is some real world value in knowing the times table (or whatever they call it nowadays) there is infinitely more value in being able to apply complex math concepts in an engineering problem. And it’s not that we weren’t doing good engineering work prior to calculators, but with calculators, we have a lot more people doing a lot more of it in dramatically shorter timeframes.

So what if we embrace LLMs and their capabilities in the same way? What does this look like?

Well, for starters (and with sincere condolonces to my students this semester) it means you can ask a lot more out of learners. If you can construct the bones of a 2-page paper on any given topic in 30 seconds or less, then I am going to ask you to do so on a wider range of more complex issues, and a lot more frequently than I otherwise would. If you can produce code that solves a simple problem with no effort, I’m going to ask you to solve significantly harder problems. I’m going to assign you to real-world, open-ended tasks that will presumably force you to research and experiment. I’m going to assume you’re using LLMs, and I’m going to not only allow it, but enthusiastically embrace it! In Data Management, for example, I’m going to not only ask you to solve a coding challenge, I’m going to give you a complex scenario that forces you to explore a challenging dataset and provide real-world value using code as the tool to explore. I don’t care if that code is generated by an LLM or something you found on Stack Overflow — as long as you’re following the instructions (and assuming I’ve crafted them well enough) you’re going to have to put in some significant effort to figure out how to apply the information you gained from those sources in a meaningful way on your data. And I get to assume that you can do a lot more of this.

Will this completely deter determined cheaters? Of course not. Someone will undoubtedly try to turn in the LLM-generated content like in my example above as their assignment, and they will receive a 0 for it when they do. (Work was too generic, did not display original thinking? Sorry, no points for you…) I will be merciless for anything that even smells like it might be plagiarized work, and I have LLMs and other tools at my disposal to help me detect it. I will expect my students to provide *real* value, *real* original thought and research into every assignment. They can use LLMs as idea creators and starter materials, and even code solutions. But in return, I expect high quality, unique results from each and every one, for each and every assignment. It’s going to be a beautiful thing. Or a terrible one, if you’re someone who isn’t used to having to perform at that level.

Welcome to the brave new world. LLMs are the new calculator, and I can’t wait to see the creativity and productivity they are going to unleash.

I’ll spare you the Star Wars video of Anakin being told he wasn’t going to be granted the title of Master, with a snarky caption of superiority. But it’s important to me for you to know that I thought about it.

Master of Science, Health Data Science

Saint Louis University School of Medicine

Get the full details (including verification steps) on my LinkedIn post.

If you’ve been with me since the beginning of this stage in my journey, I hope you’ve enjoyed the ride and gotten value from my learning out loud. What’s next? Stay tuned… :)

OK, I’m not really *back* back, but I am popping back in for a couple of relevant updates for this blog audience:

• As of last week, I have officially turned in all of my M.S. Health Data Science school work, and as of yesterday it has all been graded. I am happy to report I have officially passed! Just waiting on the transcript update.
• I have accepted a part-time adjunct faculty position with my school and will be teaching online courses in programming, data management, and data science topics for future students in the program.
• Relatedly, a rare sighting of me in a sports coat: https://www.slu.edu/medicine/health-and-clinical-outcomes-research/faculty/jason-eden.php

The world has dramatically changed since December 2022 in terms of the conversation around AI/ML. ChatGPT launched as I was finishing up my capstone project on BLOOM, and in the months that followed has taken the world by storm. I’m still thinking through what my public activity will look like (if any) post-graduation, since now there seems to be more activity — and noise — than ever before in this space. The world will look very different in six months than it does today.

On a professional note: If you have been thinking about making a serious move in this direction career-wise (and I strongly recommend at least considering it if you haven’t already), you should seriously consider checking out the online M.S. HDS program at Saint Louis University. https://www.slu.edu/medicine/health-and-clinical-outcomes-research/degrees/health-data-science-ms.php This is a program designed for health-focused careers, and as such is geared heavily towards practical application. There is still a fair amount of statistics and theory along the way, but it’s not overwhelming if you’re someone who is willing to dig in and learn. For me, the 100% online option was a major draw, but there are in-person versions of all classes as well if you prefer.

I’m still technically on a break from blogging and such, but as I warned in December, I may pop back from time to time if I feel like I can provide specific value to the overall conversation. In the meantime, I’d like to thank the organic, internal version of ChatGPT between my ears for composing this blog post for me. Not bad, although Bard probably would have been a little more concise.

The completion of my Capstone marks the last data science course in my Master’s in Health Data Science program. I have one more semester before I graduate, but the courses I have left are health research and misc. Effectively, as far as data science goes, the formal learning process I began in January of 2021 has come to a close. It’s a big moment — that’s why they call it a Capstone! This blog post may or may not be my final entry, but it marks the end of my intentional “learning out loud” formal move into the field of data science.

In The Beginning…

Nearly two years ago I published my Hello World for this blog. This was mainly a way for me to force myself to learn new things well enough to be able to explain them. To that end, this blog has been a tremendous success. As I was learning new things about data science and machine learning, I wrote about them. As I prepped to facilitate Cornell machine learning courses, school projects, and challenging work conversations, I had to dig much, much deeper, and quite often my perspective and understanding evolved along the way. I’m sure this will continue to be true as I move forward. In looking back, I’m glad most of what I wrote is at least directionally accurate. I’m sure there’s a lot I could go back and fix, but I’m happy enough with how the writing represented the journey to leave it mostly as-is, for now anyway… :)

Signal Vs. Noise

When I started out, like most people new to the field I think, I thought data science was mostly about applying statistics to real-world data and iterating over it to find patterns. In other words, I thought the secret to data science was simply having access to data and knowing how to build models. I still believe knowing how to build models is important, but I know fully realize that is a tiny, tiny, tiny little piece of what makes a data scientist valuable.

If I may opine for just a bit: There are far too many people claiming expertise around building models that don’t actually know *how* those models work, and are making really bad choices in this vacuum of understanding. A few of them have Ph.D’s and social media presences. They make what sound like completely reasonable statements that turn out to be practically wrong / incomplete. The misinformation or incomplete information from people who claim to be authoritative sources saturates the information space. It’s a dangerous realm for a newbie to step into.

One thing I really tried hard to do, even at the risk of being publicly wrong, was dig down deeply into the weeds of some very difficult topics and try to make them easier to understand. You can be the judge as to how effective I was, but one thing that really turned (turns?) me off about most data science bloggers and video makers is that they tend to repeat the same things — the high-level bullet points about whatever the latest buzzword is. I suspect very, very few of them really deeply understand — or have even tried to deeply understand — the thing that they are writing / speaking about. It’s like the marketing version of an AI expert — they can sound like the real thing, just don’t press too hard on the details. We all start somewhere, but it’s that laziness — the jumping on the next bandwagon, the not even trying to explain a concept topic and just hoping the audience doesn’t actually ask you about it — that really got to me. Thankfully, if you’re patient you can usually find at least one person who understands it well enough and, if you’ve done enough homework, has explained it well enough that you can take it to the next level of understanding. And when you find those folks, you bookmark them and read everything they put out. But it’s a lot harder than it should be to find the good stuff. There’s an overwhelming amount of noise (pun intended). And as this field gets more popular, it just seems to get worse.

Real-World Consequences

It’s disheartening, and is starting to cause people who are good at this to reconsider the effort and risk. When the world can’t tell the difference between a true expert and a charlatan, and indeed, believes the charlatan to be better because of amount of work produced or promises claimed to be delivered on, there are probably more rewarding things to do with your time. Good people are leaving the field because transparency and honesty are penalized in the marketplace far too often. And that’s tragic.

Real Science

That said, there is an immense amount of opportunity in this field. I consider some of my frustration to just be growing pains of a new, difficult field of study that suddenly got very popular. Looking ahead, as data volume explodes and AI models get better and better at ever more complex tasks, the sky is the limit in terms of how machine learning can impact human lives (and hopefully for the better.) Just like every other scientific field, there are risks, but there are also rewards. Figuring out how to leverage the power of these things we are building while making them safe and universally beneficial is going to take a lot of care.

And it’s that part of it that gets me most excited. This is not a solved problem. There are virtually limitless opportunities, both extremely simple and terribly complex, to find ways to use data science for good. Everyone wants to predict the stock market, at first (and the first ones to crack that nut will probably be the first and very last gazillionaire’s…) But the things that really matter — early detection of cancer from medical imaging, helping a retiree manage limited finances, helping a struggling student understand math — all of those are things that AI could potentially be applied to solve for. We have the technology. We just have to figure out how to use it.

My Next Journey?

This is far and away not the end of my learning curve in data science. I may well write a new blog post from time to time to support something I’m doing in an academic or professional sense, but a lot of the next phase of my learning will be application-focused. How do we move forward? How do we employ these technological advances into the areas where they can benefit humanity the most? (Ahem… healthcare?) What can we do to ensure that people coming into this field step on fewer landmines than I did getting started in terms of finding good, accurate, valuable information? How do we democratize AI so that it benefits everyone, not just the largest companies with the most resources?

I still love my job and have no intentions of changing careers in the near-to-medium term, but I do intend to look for ways to contribute both within and outside of my main job responsibilities. I don’t need to exclusively be an AI expert — my cloud computing knowledge still matters, as does the learnings from my M.B.A., my psychology degree, and my technical education and IT background. The relatively unique makeup of what I know means I should be able to find unique ways to leverage any number of technologies — including AI/ML — into any number of situations for a greater good / better experience / efficiency gain / and so on. In some respects, I just have to make sure I’m being a good steward of the time I have and the opportunities that will be available to me.

If only I had an AI model to optimize that…

Thank You

So goodbye for now (and thanks for all the phish — or, rather, the lack thereof.) Thank you for reading, clapping for, and commenting on my work. It was mostly for my own benefit, but I’m glad it did at least a few people some good along the way. Perhaps I’ll find a reason to pick up the pen here again, or maybe I will be doing something completely different this time next year. Either way, it’s been a rewarding journey, and I hope you’ve enjoyed being part of it. See you on the flip side!

For the last couple of months I’ve been too busy to write about the things I’ve been learning and doing, but if you’re curious (or have a previously incurable case of insomnia) click this link to access my just completed Capstone project for my Master’s in Health Data Science Program.

If you’re more of a slides / presentation style person, you can click this link and get an overview in slide form.

Tl;dr — I spent a semester learning about NLP and LLMs (some of that learning I’d already blogged about) and then tried to build out a process to customize a large language model for question answering — notably, a model that when I started the program did not have the native capability to be configured as such. I produced the first published BLOOM LLM for question answering tuned for the SQuAD dataset, which can be accessed here:

jasoneden/bloom560m-squad-helloworld · Hugging Face

There’s a lot of content and context herein, and I may go back and blog about some of the many, many things I have learned since my last post, but for now I’m just going to celebrate the completion of this milestone and step away from the computer for a bit. :) In the meantime, I look forward to any comments from anyone regarding the project, what I got wrong, what I got right, and ways I could have done it better.

In a couple of previous posts I’ve talked about the level of data crunching that has to happen in order to train a Transformer Large Language Model (LLM). This turns out to be an exceptionally expensive proposition when starting from scratch. The cost can easily run into the 8–9 digit range (or more) and in addition there are environmental impacts for each new model trained. Thus, the ability to take a large LLM that someone else has already paid for (and whose environmental damage has already been done) and customizing it for your own purposes seems like the financially smart and environmentally friendly thing to do.

And yet, things are never quite that simple in the real world, are they?

Garbage In, Garbage Out

One of the problems encountered when trying to build one of these massive LLMs is simply gathering enough data. As a result, models are frequently based on a corpus that includes a significant amount of uncurated, publicly available content. While this succeeds in helping LLMs be able to make predictions and generate text in response to prompts, sometimes the responses are… shall we say, suboptimal?

For example, you may be surprised to learn that there exists a significant amount of text on social platforms today that is represents biased and bigoted worldviews. Models built using this data as the source for predictions can sometimes return some very atrocious responses to certain prompts. However, even models like BLOOM, where a full 2/3 of the data used to build the model were hand-picked by researchers in an attempt to mitigate such issues, still hold inherent human biases in their text. After all, even writers with the best of intentions (myself included) can have blind spots to social/cultural biases that can produce harmful responses based on the context in which a question is asked.

To illustrate this (in a very, very tame way) I updated the notebook where I first tested the small BLOOM model and prompted it with two very similar phrases: “The ______ works as a” — and set it to predict the next word. The blank was either “man” or “woman.” Otherwise, the prompts were identical. The next text generated across multiple trials was reflective of gender biases.

Thus, both obvious and inherent/hidden biases on the part of the writers of the text that is used to build a LLM will predictably become part of the response patterns the LLM will generate to the prompts it is given. It is very difficult to mitigate, if it’s even possible at all, in the model-building phase. This raises questions as to whether any generative machine learning model will ever be able to overcome this problem, at least given current data and technology available.

Fine Tuning

If we assume that there are ways to mitigate this issue (something I’m noodling on and may write more about in the future), then the power and potential of generative LLMs becomes enormous. But how does this process work?

If you think back to neural networks (and Transformers are just a fancy/advanced version of neural network architectures) then you will recall that the output of training a machine learning model is simply a set of weights that have been iteratively adjusted until they meet the necessary performance criteria. As such, it is also appropriate to think of a pre-trained Transformer model as simply a set of weights (a complex set of weights with some very specific rules about how it works with data to make predictions, but nonetheless just a bunch of weights.) It is relatively straightforward, then, to take a new corpus of data — say, data present on your company’s website, knowledge base articles, and other useful things — and see how good that pre-trained model with those weights does at predicting things within that dataset.

Odds are, the predictions will be pretty good, but not optimal, since your domain-specific knowledge likely did not exist in the training data for the LLM. However, just like when the original model was built, you can use back propagation (gradient descent) to iteratively adjust the model weights such that it performs better and better on your own data. Now the resulting model might not perform quite as well on general purpose prompts as it did before you tuned it, but if you’re building the model for a specific purpose — maybe a chatbot that answers HR or policy questions, locates documents of interest, or similar — then you’re typically not concerned about how the model performs for out of domain tasks. The fact that it *can* answer questions or make text predictions that fall outside of your domain area is just a bonus. What you really care about is having a model that is extremely conversational, and when it comes to the kind of things your users typically need to know, extremely knowledgeable and accurate.

LLMs present the greatest opportunity to date in terms of building these kinds of conversational AI systems that can seem to flex and intuit meaning based on context the model builders did not anticipate, and thus mimic human conversation to ever greater degrees. The possibilities for good are virtually limitless. We just have to figure out how to keep them from saying things that might get a human counterpart rightfully fired.

That is not going to be an easy thing to do when basing your predictions on writings that contain the thoughts and predispositions of human beings.