With the increasing number of online privacy laws such as the EU General Data Protection Regulation (GDPR) and the upcoming California Consumer Privacy Act of 2018 (CCPA) there is increased liability associated with storing Personally Identifiable Information (PII).
Introducing Media Service Early Access
Media Service enables you to send transactional emails to customers without having to store their email addresses, reducing the liability involved with storing Personally Identifiable Information (PII).
Now, with the Media Service, you no longer need to store sensitive email addresses yourself. The Media Service will store the email address on your behalf and instead will grant you a capability to send email to the address without exposing the address to you. You can store and manage these capabilities within your systems without fear of leaking sensitive data.
Using Media Service
With the Media Service, you exchange your customer’s email address for the capability to send email to the customer. This frees you from having to store sensitive PII while still being able to communicate with your customers via email.
To learn more, see the Media Service documentation.
Pricing and Availability
During Early Access, Media Service is free to use. Once Early Access ends, you will pay only for what you use.
To sign up, contact email@example.com.
With the arrival of HTTPS Everywhere, all browsers now mark non-HTTPS traffic as insecure. However, management of Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates that enable HTTPS can be quite complicated.
Introducing Certificate Manager Service Early Access
Certificate Manager Service simplifies public Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificate management while letting you deploy those certificates wherever you like.
Certificate Manager Service lets you request a publicly trusted certificate and have that certificate delivered to your storage solution so that your systems can distribute and use it as required. Your certificate is not locked into use with only Capability Services.
Online delegation of authority looks nothing like how we delegate authority in our every day lives. The prevalence of Access Control List (ACL) authorization pattern (list of permissions attached to a resource) has constrained many systems to centralized access control. While, for some highly sensitive use cases, centralization is desired, for others, the use of ACLs leads to overconstrained solutions.
Introducing Membrane Service Early Access
Membrane Service solves the problem of being able to delegate authority in a decentralized manner. It does this by using capabilities and offering capability-based authorization at any scale in accordance with any policy via membranes.
This post explores why the speed of software development can slow to a crawl inside large organizations. In particular, we will consider software services and their customers from the perspective of resource contention. It turns out that this framing highlights a major contributing factor that slows software development. We will conclude with what can be done about it.
Consider the problem of resource contention. Software services require resources to operate, and those resources are finite. When users use a software service, resource contention problem occurs as soon as there is more than one user. The key question to consider is, who is responsible for managing the resource contention problem? We have two possibilities. The service itself can manage the problem in a centralized way or the users can manage the problem in a decentralized way. This is the key insight to extract from this framing.
The service itself can manage the problem in a centralized way or the users can manage the problem in a decentralized way.
The resource contention problem can be managed by the service or by the users. In the case of the service, the management can be centralized within the service. In the case of the users, the management must be distributed.
There is a cost associated with managing the resource contention problem.
When the service bears the cost, it bears the entire cost of managing the problem. A typical solution to the resource contention problem is for the service to become multi-tenant. To every user, it seems as they are the sole tenant of the service and need not worry about managing resource contention.
When the users collectively bear the cost, the cost for any single user is mostly small, most of the time. A typical solution takes the form of an ad hoc, distributed, and probably incorrect, consensus protocol. This protocol is typically referred to as “careful” or “being a good citizen.” Every user knows that there are other users who at any point can do something that severely degrades their own use of the service, or worse, they’re unaware of it. Typically, users coordinate with each other in order to manage the shared service and not exhaust its resources.
Consider what happens when a user uses multiple services for which they have to manage resource contention with other users. Every new service requires adoption of a new, ad hoc, distributed, and incorrect, consensus protocol. While the cost of coordination may be low initially, the cost, per user, grows super-linearly with the number of services used, since all of the users of each new service, some of them unknown, must be coordinated with.
Alternatively, when each service manages the resource contention problem by offering multi-tenancy, every new service requires no coordination from the user. The cost, per user, increases linearly with the number of services used and is limited to learning how to use the service.
When each service manages the resource contention problem by offering multi-tenancy, every new service requires no coordination from the user.
Another relevant concept in the dynamic between services and users is the notion of internal and external users.
Users external to the business, tend to have other options, and therefore are less likely to put up with additional cost of coordination required for using the service. On the other hand, internal users typically have no such luxury, are cost insensitive, and must use the service designated for them. This tends to lead to a pathology where the service seems justified in not paying the additional cost of providing multi-tenancy, as the internal users have to use the service no matter the cost of coordination. But, what is often missed, is the super-linear cost the internal users must bear for their distributed management of resource contention. Fortunately, there are things we can do to avoid this situation.
Service seems justified in not paying the cost of providing multi-tenancy, as the internal users have to use the service no matter the cost of coordination.
We can demand that all services manage the resource contention problem by being multi-tenant. Alternatively, we can ensure that internal customers have a choice of using or creating another service that is multi-tenant, which may eventually lead to multi-tenant services as the cost of their adoption is lower.
Lastly, I want to highlight a pathological move not to make.
Forcing the use of a single service is the right move if and only if that service is multi-tenant.
There is great efficiency to be gained by removing duplication of effort. Forcing the use of a single service is the right move if and only if that service is multi-tenant. Otherwise, the organization is placed in a configuration where internal users must bear the cost of managing resource contention and cannot improve their situation by using a multi-tenant alternative.
What is a type?
Something, something, computer programming…
What I find interesting about types is that they enable me to think in terms of patterns as opposed to in terms of specific examples. A type system, allows me to express something in terms of patterns, and the patterns can be arbitrarily abstract. Ironically, let’s look at some examples of patterns, i.e. types.
A Unit type can be thought of as a pattern of “something”. It conveys the notion that “something” exists. There is an instance of “something”.
It may help to think in terms of receiving an email in your inbox, but that you only saw the number of new messages increase by one. You haven’t read the email. You know nothing about it. You just know that you have another email in your inbox. That’s like Unit type. It conveys that “something” (some email) exists.
I tend to think of Unit type as a “signal”. If you imagine a light switch, a “signal” is not whether or not the light is on or off. A “signal” would be the flipping of the switch, the flip itself. Imagine you can’t see the light, you just hear the switch flipping. flip flip flip… three signals, three Unit types.
Ok, that might still be fairly abstract. Let’s contrast this with something more familiar, but let’s name it something really weird, like, the Sum type.
A Sum type can be thought of as “exclusive or” pattern. In other words, it can be this “something”, or that other “something”, but not both. For example, consider the notions of True and False. We say that something can be True or False but not both.
In fact, True or False (but not both) is of the type Sum with the shape of Unit + Unit. “Unit + Unit” means that the Sum type has space for two Units, but the fact that it’s exclusive or, means that it will only accept one Unit. True is defined by putting “something” (of type Unit) into the first space of Unit + Unit. False is defined by putting “something” (of type Unit) into the second space of Unit + Unit. What if you want to put “something” into both? You can’t, because by definition, we say that you can only put “something” into one of the spaces. Why is True putting “something” into the first space and not the second? The answer is that that’s the convention that most people who use Sum types use. You can use any convention you want, but it may be more difficult to understand what you’re communicating.
Remembering that Sum type describes a pattern of “exclusive or” helps me to remember how it works.
Going back to our light switch example, and to illustrate the difference between Sum and Unit, imagine that we now can tell whether the light is on or off. We can represent the pattern of knowing whether the light is on or off by Sum type with the shape Unit + Unit. If light is on, we will put “something” into the first space. If light is off, we will put “something” into the second space. It can’t be both on and off. All we need to put into one of the spaces is of Unit type, a “signal”. Remember that it is not the “signal” that tells us the light is on. The space the “signal” is in is what tells us whether the light is on or off (first space means light on, second space means light off). The nature of the “signal” itself is immaterial, we only care that “something” is in the space.
Why is it called “Sum” (as in “summation”) type? The name comes from how one would calculate the number of unique things that one can represent using a Sum type. For example, the Sum type Unit + Unit, can represent only one plus one, that is, two things. This is why it’s used for representing True and False, as those are exactly two things. If, for some reason, we wanted to represent four things, for example: Spring, Summer, Fall, Winter, I could use a Sum type of Unit + Unit + Unit + Unit. One plus one plus one plus one is four. And a season can (for our illustration purposes here) be either Spring, or Summer, or Fall, or Winter, but not more than one of those.
A Product type can be thought of as “and” pattern. In other words, it can be this “something” and that other “something” together.
A Product type that has two “somethings” would be Unit x Unit. “Unit x Unit” means that the Product type has space for two Units (“Unit x Unit x Unit” would mean that the Product type has space for three Units). For example, a weekend is Saturday and Sunday. We can represent Saturday by putting “something” into the first space and Sunday by putting “something” into the second space. Now, this is a somewhat not useful example of a Product. Let’s come up with a better example.
Remember our Sum type of Unit + Unit where we defined True and False? Let’s name that particular Sum type shape of Unit + Unit a Boolean type (it’s named after George Boole). Now that we have our Boolean type (which represents the notions of True and False), let’s define a more useful Product of the shape Boolean x Boolean. “Boolean x Boolean” means that the Product type has space for two Booleans. We’ll still think about the weekend, but this time, the first space will represent whether we are working on Saturday, and the second space will represent whether we are working on Sunday. So, if I’m working on Saturday and Sunday, I would represent that as True x True. If I’m working on Saturday, but not working on Sunday, I would represent that as True x False. Not working on Saturday, but working Sunday would be False x True. And, lastly, not working all weekend would be False x False.
Why is it called “Product” (as in “multiplication”) type? That’s because to calculate the number of unique things that one can represent using a Product type, we multiply the number of things that can be in the first space by the number of things that can be in the second space and so on. Notice, in our weekend representation of Unit + Unit, we could only put one thing in each space (Saturday and Sunday), so the number of things we could represent was one times one is one, the weekend. However, once we could put two things into each space, as in our example of whether we are working on the weekend, we could put two things into first space (True, False), and two things into second space (True, False). Two times two is four, and the Product type of Boolean x Boolean could represent four different work schedules over the weekend.
A Void type can be thought of as a “nothing” pattern. This pattern is either obvious to people, or very difficult to understand.
In the email inbox example, a Void type means that an email hasn’t arrived. You received no signal, “nothing” happened, no change at all.
In the light switch example, a Void type means that you can’t see if the light is on or off, and you can’t hear the flipping of the switch. It’s not that you will eventually hear or see something, but not yet. It’s that you will never hear or see anything. “Nothing” will happen. Void is the absence of any signal.
We now have some understanding of other types that can help us understand the nature of Void type. Imagine I have a Sum type with the shape of Void + Unit. “Void + Unit” means that the Sum type has only one space, and it is only the second space. There is no first space in Void + Unit type. How many things can you represent using Void + Unit type? It is zero plus one. You can only put zero things into the first space, because there is no first space. There is only second space, into which you can put one thing. Void type is analogous to zero.
To see this another way, consider a Product type of Void x Unit. How many things can you represent using Void x Unit type? It is zero times one, which would be zero. The first space doesn’t exist, it is of type Void, and therefore we have no way of constructing something that fits the pattern of “nothing and something”. The problem is that we cannot construct a something that fits the pattern of “nothing”, so we can never construct a something of Void x Unit type.
An arrow type is a pattern of “how things on the left side of the arrow relate to the things on the right side of the arrow” (but not the other way around). This sounds fairly abstract, let’s dive into an example.
Previously, we used the example type Product of Boolean x Boolean to describe a weekend work schedule. Let’s call this Product type a Schedule type. To build our example, we’ll also consider a Sum type of Unit + Unit, where putting Unit into the first space will mean worker Tristan, and putting Unit into the second space will mean worker Dale. Let’s call this Sum type a Worker type. Now, we can describe an Arrow type of Worker -> Schedule which is a pattern of “how Workers on the left side of the arrow relate to the Schedules on the right side of the arrow”.
Other common names for Arrow type are Exponential type, or Function type. The reason for “Exponential” name, is the same as for Sum and Product types, that is, it describes a way of how to count how many number of unique things one can represent using an Arrow type. Remember that our Arrow type is Worker -> Schedule. Worker type is a Sum type of Unit + Unit, which can represent, one plus one, so two things. Schedule type is a Product type of Boolean x Boolean, which can represent, two times two, so four things. The Arrow (“exponential”) type can represent Schedule ^ Worker number of things, or four to the power of two things, so 16 things. Let’s count them:
- Tristan -> (True, True), Dale -> (True, True)
- Tristan -> (True, True), Dale -> (True, False)
- Tristan -> (True, True), Dale -> (False, True)
- Tristan -> (True, True), Dale -> (False, False)
- Tristan -> (True, False), Dale -> (True, True)
- Tristan -> (True, False), Dale -> (True, False)
- Tristan -> (True, False), Dale -> (False, True)
- Tristan -> (True, False), Dale -> (False, False)
- Tristan -> (False, True), Dale -> (True, True)
- Tristan -> (False, True), Dale -> (True, False)
- Tristan -> (False, True), Dale -> (False, True)
- Tristan -> (False, True), Dale -> (False, False)
- Tristan -> (False, False), Dale -> (True, True)
- Tristan -> (False, False), Dale -> (True, False)
- Tristan -> (False, False), Dale -> (False, True)
- Tristan -> (False, False), Dale -> (False, False)
The reason for Arrow type to be called Function type is that Arrow type corresponds to what people mean by “function” in mathematics. If I have a thing of Arrow type, for instance, Tristan -> (True, True), Dale -> (True, True), then if I want to find out Tristan’s schedule, I would provide Tristan as input to the function, and the function would return the result (True, True).
Why call it an Arrow type then? There is a thing in mathematics called “up-arrow notation”, and it so happens that a single up-arrow in up-arrow notation corresponds to “exponential”. Discussing multiple arrows is out of scope of this post, but mentioned here for the curious.
A Value type can be thought of as a pattern in contrast to the Unit type pattern. Where Unit type was a “something” pattern, Value type is “this particular thing” pattern.
In the email inbox example, again, by contrast, where Unit type would be a signal that some new email arrived and we only care about the signal. Value type would be saying that a particular email arrived, and while we care that email arrived, we also care about the value, the particular contents of that particular email.
Another way of phrasing this, is that for a Unit type we only care about the signal. In the sentence “This thing exists”, what we focus on in Unit type is exists. For Value type, we focus on the entire sentence this thing exists, because we are trying to express the pattern that particular thing not only exists, but that it is a particular thing.
For example, think of the boolean True. Looking at True through the lens of “this particular thing”, we care that it is True, and that it is not False. The value of True is True.
Time to get weird.
Type type expresses the pattern of “a pattern” 😬. We covered multiple examples of Type type. Unit is of type Type. Sum is of type Type. Product is of type Type.
There is an important concept to highlight. Earlier, we defined True and False as being of the type Boolean. What’s worth highlighting is that Boolean is of type Type, but True is of type Boolean.
Also, notice that the type Type is of type Type. This is because the pattern of “a pattern” fits the pattern of being “a pattern”.
Everything is a Value
Recall that when we talked about the Value type, we were expressing the pattern of “this particular thing”. If we have the type Boolean, and we have a particular boolean, say True, then the particular boolean True is of type Boolean, but it is also of type Value. This is because the boolean True fits the pattern of “booleans” and it fits the pattern of “particular thing”. “Fitting a pattern” is referred to as “inhabiting a type”. So, the boolean True inhabits the type Boolean and it inhabits the type Value. This is because it “fits the pattern of booleans” and it “fits the pattern of particular thing”.
Types are Values and Values are Types
Types are Values. This is because Type type (a pattern of “a pattern”) fits the pattern of being “a particular thing”. The type Type inhabits the type Value.
Values are Types. This is because Value type (a pattern of “a particular thing”) fits the pattern of being “a pattern”. The type Value inhabits the type Type.
Everything inhabits Unit
Notice that the Unit type is the pattern of “something”. This means that everything that exists fits the pattern of being “something”, therefore everything that exists inhabits the type Unit.
It is worth highlighting the interplay of Void type and Unit type. The Void type itself inhabits type Type, inhabits type Value, and inhabits type Unit, because the pattern of “nothing” fits the pattern of being “a pattern” (Type), fits the pattern of being “a particular thing” (Value), and fits the pattern of being “something” (Unit). However, notice that there is nothing that can inhabit the type Void. This is because to fit the pattern of “nothing”, there can be nothing there. If there was something there, it wouldn’t be nothing.
That’s all for now…
Let’s stop before it gets weirder (like thinking about Arrow types with Void types), but this should be a fair introduction to the basic concepts with hints at where things start to get out of hand and we might need something more sophisticated than the english language.
While what I’ve described here (in english) is a description of a type system, there are different type systems that can be described, and they can differ from each other in subtle ways. The differences between them doesn’t make any of the other systems incorrect, nor does it make this description correct. But, hopefully, I managed to communicate some intuition about one kind of type system to you.
If something isn’t clear, please comment/respond and we’ll talk about it.
Also, thank you Dale Schumacher for pointing out errors in the early drafts and thinking through all this stuff with me.
If you’re using a Kanban board to visualize your work, in its most general form, it probably looks something like:
It turns out that this is exactly backwards with how we intuitively visualize time. As Jabe Bloom points out, the “set of columns reflects an inversion of our innate understanding of the flow of time.”1 Instead of time flowing from left to right in a Past, Present, Future order, it is inverted, and on a Kanban board it flows from right to left.
Contrast this with what a Kanban board would look like if it was coherent with how we usually think about the time arrow.
Once I rearranged a Kanban board this way, I have a hard time thinking of it differently. There are multiple things that become coherent with this arrangement.
- The new arrangement is coherent with how left-to-right readers visualize the arrow of time. By coherent, I mean a very abstract coherence in the sense that “Happy” is coherent with “Up” or that “Sad” is coherent with “Down”.
- When “walking the board”, we are taught to walk “backwards” from DONE to TO DO. Well, in this new arrangement, there is nothing backwards about it. Walking the board becomes coherent with the board arrangement and the flow of time.
- The cards end up traveling from right to left. For some reason, that is more coherent with “pull”. To contrast with a standard Kanban board, cards traveling left to right seems to me more coherent with “push”.
- Looking at “TO DO” column in the future and on the right, feels more coherent with “TO DO” being our vision of the future that we are “pulling” into reality one card at a time. It also, to me, highlights better my opinion that a backlog is just a place where everything goes stale without us worrying about it.
- The clutter of a “TO DO” column seems easier to dismiss when it’s on the left. When it’s on the right, the clutter of “TO DO” goes from “we have a lot of work to do” to “we have no coherent view of what we want in the future.” The difference is very subtle, but I think it’s there.
Will changing your Kanban board this way make you 50% more productive? No. However, while I see no compelling reason for the predominant TO DO, DOING, DONE arrangement, there seems coherence to be gained by switching to DONE, DOING, TO DO.
1 Bloom, Jabe (2012). The Moment of Pull – Meditations on time and the movement of cards. Retrieved 9 Feb 2018.
2 While this Kanban board arrangement came to me while reading Jabe’s “The Moment of Pull,” it is not a new idea. For example, see: Rybing, Tomas (2015). Mirrored Kanban Board. Retrieved 9 Feb 2018.
After writing the original How Long Will It Take? post, I kept wondering how to measure if the estimation method described therein (from here on referred to as “Historical Lead Time”, or HLT) is effective, for some definition of effective. What I realized is, since the estimation method does not require human input, I could use historical data and simulate what the method would estimate at any particular point in time. This blog post describes this experiment and demonstrates a very surprising finding.
TL;DR: It turns out that the HLT method minimizes estimation error better than every other tested method except one, which is… *drumroll*… “pick the average so far” as the estimate. Read below for details and caveats. Also, it would be very helpful to run this experiment on many data sets instead of just the one I used, please contact me if you can provide a data set to run this experiment on.
Determine usefulness of estimating software work using percentile estimates based solely on observed past data as described in How Long Will It Take? .
HLT estimates are better than random estimates. (Spoiler: they are! … or more correctly: experiment results do not refute this hypothesis)
Sum of square error, which will be the difference between estimated duration and actual work item duration, squared.
The experiment is a simulation of what would the estimates be at specific times in the past. Given a data set of work start and stop times, simulation starts after completion of first work item and ends after completion of last work item in the data set.
Experiment uses multiple estimation models. The model with the least cumulative sum of square error is deemed the best. Where appropriate, models are tracked per 25th, 50th, 75th, 90th, 95th, and 99th percentiles. Models used in the experiment are:
- HLT: Estimation method described in How Long Will It Take?.
- Levy: Estimation method that assumes distribution of observed work item durations can be described as a Levy distribution. This model is included to showcase a terrible model.
- Gaussian: Estimation method that assumes distribution of observed work item durations can be described as a Gaussian/Normal distribution. This model is included to showcase a “dumb” model as a sanity check.
- Random: Estimation method that simply picks a random number between zero and longest duration observed so far. This model is included to provide a baseline to compare against.
- Weibull: Estimation method that assumes distribution of observed work item durations can be described as a Weibull distribution. This model is included because it is seems to be the go-to model used by people who take estimation seriously.
In addition to the above models, each model is also tested with and without sample bootstrap to a sample size of 1000, as described in How Long Will It Take?
Data set used for the experiment is Data Set 1, consisting of 150 work item start and stop times.
Each simulation is performed using the following procedure:
- Using Data Set 1, create a timeline of start and stop events to playback.
- Playback the timeline created in 1.
- Upon observing a work item start event, notify the estimation model of work item start. If the model can generate an estimate (model must observe two completed work items prior to generating an estimate), compare the generated estimate with actual known duration, calculate the square error, and record it.
- Upon observing a work item stop event, notify the estimation model of work item stop.
- Continue playback until the timeline is exhausted.
A rich way to demonstrate the results is to plot the cumulative sum of square error for each model and each percentile together (model+percentile, e.g.: “levy 0.99”, means Levy model 99th percentile, “hltbs 0.75”, means HLT model with bootstrapped sample at 75th percentile). These plots are included below. In consecutive plots, the worse performing model+percentile line is eliminated so that we can see more detail regarding better performing models. Also, the shape of error accumulation is instructive. The elimination order is (from most accumulated error to least accumulated error):
levy 0.99, levybs 0.99, levy 0.95, levybs 0.95, levy 0.90, levybs 0.90, levy 0.75, levybs 0.75, wbulbs 0.99, hltbs 0.99, wbul 0.99, hlt 0.99, gausbs 0.99, gaus 0.99, levy 0.50, levybs 0.50, gausbs 0.95, gaus 0.95, wbulbs 0.95, rand, wbul 0.95, gausbs 0.90, gaus 0.90, hltbs 0.95, hlt 0.95, levy 0.25, levybs 0.25, gausbs 0.75, gaus 0.75, wbul 0.90, wbulbs 0.90, hltbs 0.90, gaus 0.25, gausbs 0.25, wbulbs 0.25, wbul 0.25, randbs, hltbs 0.25, hlt 0.25, wbulbs 0.50, wbul 0.50, hlt 0.90, wbulbs 0.75, wbul 0.75, hltbs 0.50, hlt 0.50, hltbs 0.75, hlt 0.75, gausbs 0.50, gaus 0.50
Here is a list that performed worse than “rand”:
levy 0.99, levybs 0.99, levy 0.95, levybs 0.95, levy 0.90, levybs 0.90, levy 0.75, levybs 0.75, wbulbs 0.99, hltbs 0.99, wbul 0.99, hlt 0.99, gausbs 0.99, gaus 0.99, levy 0.50, levybs 0.50, gausbs 0.95, gaus 0.95, wbulbs 0.95
Here is a list that performed worse than “randbs”:
levy 0.99, levybs 0.99, levy 0.95, levybs 0.95, levy 0.90, levybs 0.90, levy 0.75, levybs 0.75, wbulbs 0.99, hltbs 0.99, wbul 0.99, hlt 0.99, gausbs 0.99, gaus 0.99, levy 0.50, levybs 0.50, gausbs 0.95, gaus 0.95, wbulbs 0.95, rand, wbul 0.95, gausbs 0.90, gaus 0.90, hltbs 0.95, hlt 0.95, levy 0.25, levybs 0.25, gausbs 0.75, gaus 0.75, wbul 0.90, wbulbs 0.90, hltbs 0.90, gaus 0.25, gausbs 0.25, wbulbs 0.25, wbul 0.25
Here is a list that performed better than “randbs”:
hltbs 0.25, hlt 0.25, wbulbs 0.50, wbul 0.50, hlt 0.90, wbulbs 0.75, wbul 0.75, hltbs 0.50, hlt 0.50, hltbs 0.75, hlt 0.75, gausbs 0.50, gaus 0.50
The vertical axis is the accumulated square error with units and values omitted as relative comparison is sufficient. The horizontal axis is enumerating estimates from first to last. Note that the same line style and line color does not represent the same model+percentile from plot to plot. Refer to the legend for identification of model and percentile.
Main concern is that experimental data is only one data set of 150 items. While the results are surprising, they may not be typical. I need other data sets to run this experiment on (please get in touch if you’re interested in testing your data set).
Regarding accuracy, the fitting of Levy distribution was fairly unsophisticated (calculate mean and variance of sample and use that to generate a Levy distribution). I didn’t expect Levy to perform well and as it was just background to testing the main hypothesis, I didn’t bother implementing a more sophisticated distribution fitting. Weibull distribution, on the other hand, is a pretty good fit as it uses least-squares fit to observed distribution. In summary, Levy distribution fit is crap, Weibull distribution fit should be pretty good. Gaussian distribution fit is straightforward, so it should also be good.
It is interesting to see the impact of outliers on estimation method (large spikes in error graphs). While an outlier destroys some estimators (one can observe points in the graphs where estimator makes a turn for the worse and rapidly departs from best performer), other estimators seem to be robust to outliers. Note that model may be robust or not depending on which percentile is used for estimation.
Another thing of note is that the best performing percentiles are 50th and 75th and not others. This attraction toward the average was a surprise.
Why does “pick the average so far” (more precisely, pick 50th percentile of estimated normal distribution from observed data without bootstrapping) work so well? I assume that part of it is due to normal distribution being robust to outliers, especially once there is enough data to anchor the distribution away from the outlier pretty well. I’m not sure why HLT 75th percentile is better than HLT 50th though.
Bootstrapped random estimator (rndmbs) performed really well, and it also has the same shape as the winning estimators. Note that rndmbs used the same random seed as rndm to select a number between zero and maximum in the sample. What most likely happened is the interplay between bootstrapping process (sampling with replacement) and random estimate being a random number between zero and maximum in the sample. Early on, before the outlier, bootstrapping did not change the maximum, and random estimators chose the same number up to the same maximum. Once an outlier occurred, we see that rndm selected it at least twice. However, it may be the case that the bootstrapping process for rndmbs did not pick the outlier into the bootstrapped sample, allowing rndmbs to pick from a smaller numeric range. From the plot of rndm, it looks like rndmbs only had to get lucky like this twice.
It seems that in order to minimize error in estimates, the best thing to do is pick an estimator robust to outliers. In particular, the best (according to this data set), is to estimate a normal distribution from observed data and pick the mean. If this holds for other data sets, it means that we can all let go fancy statistical methods and use this very simple “pick the average” approach from now on. Imagine how much simpler our estimating lives could be ;).
Questions For The Future
Do these patterns hold for other data sets? If you have a data set, please get in touch.
The experiment only checks the estimation at the start of work (typically when we do estimation), but this doesn’t take into account the full HLT technique of continuous estimation. How good would these estimators be in a continuous (for example, once a day) estimation?
The experiment does not check if models get better at estimation as time progresses. This may be interesting to see.
Typically, when we estimate, the impact of finishing early and finishing late is asymmetric. What would be the results under different penalties for having estimates that are too optimistic (work actually takes longer than estimate).
What is the impact of choosing different seeds for bootstrapping as well as different seeds for estimators using random?