People like an indication of costs before starting a project, and rightly so.
I'll show here how to estimate software projects, and create a budget. We'll create a concrete template which can be used for project proposals before work has begun, or as an estimation template you pass out to agencies.
Isn't this waterfall?
It can be, but it depends on how you utilize the estimates. Preferably you'd just see it as a likely predicion, not a strict plan. In most software projects you'll need to adapt as you receive new information. You should be able to add something new along the way, but have to drop something else if you want to stick within the limits of a budget.
If you have your own team and want to predict release timelines you are likely better off using empiric estimation methods based on measured velocity you can always do the initial discovery part of the project as a separate mini-project before starting implementation.
Firstly - estimations can only be as good as the specifications, which don't necessarily exist. Ideally there has been some sort of workshops, design sprints or other forms of product discovery before attempting estimation.
If not and you're still struggling to figure what it is this project is about, then it's too early to do any sort of estimates since you have no scope. You can always do the initial discovery part of the project as a separate mini-project before starting implementation.
If you have something concrete enough to estimate, then following the points below you'll stand a fighting chance.
1. Figure out as much as possible about the scope within reason
Discuss with the stakeholders you have access to. Obviously you cannot achieve an exact specification. The only way to do that is to implement the whole project, then travel back in time giving yourself the outcomes. To avoid any time-space paradoxes just try to clarify everything possible within reason.
You will be constrained by time, access to stakeholders and certain things simply not being decided on yet. If there are known unknowns, then document them as known unknowns.
2. Make and state your assumptions.
One can't really estimate unknowns, but you can make them concrete by making likely assumptions, and then clearly stating those assumptions in the context of your estimate. E.g. you could assume that the CRM used in some solution is going to be HubSpot, even though it is not decided on yet.
3. Split the work into estimateable chunks.
This can be e.g. user stories, but if many people are involved in a story then you may want to split it further into sub-tasks. If chunks end up being over 10 days of work, it likely contains a lot of uncertainties.
That may be ok depending on the detail wanted. Sometimes these are fuzzy things in the far off future and we're forced to do placeholder estimates to stipulate that "this will cost something".
4. Phase the work
The bigger and longer the project, the more innaccurate the initial estimates will be. Sometimes stakeholders will want a total project budgets even though the latter needed features are far off and poorly specified.
It helps to phase the work so estimates for earlier phases can be considered independently from the latter fuzzier phases. After the first phase you can start using to empiric measurements of development pace to make the predictions more accurate (See e.g. Mike Cohn's Agile estimation and planning).
5. Make sure all the of the boring stuff is included
It's easy to get caught up in features and not consider setting up environments, continous delivery and other automation to enable you to get these features into customer hands.
It's also good to include various project management tasks and likely total time spent in iteration meetings (plannings, retros, demos and other meets where all team members are usually present). These eat up sizeable chunks of your budget although necessary.
6. Try to think of all the common requirements for a certain task type
It's easy to forget common requirements which stakeholders don't think of mentioning explicitly, leading to consequential underestimation of work.
For form inputs: Character limits, default values, input masks, validation messages, server side checks for duplicate username etc.
Graceful user friendly error handling
Mobile layouts / Responsive design
Immediate user feedback on various actions
Browser compatiblity requirements
7. Let the people who are going to do the work do the estimation
If you don't have access to them (e.g. there is no team yet), then the best you can do is find a person of similar profile and ask them to estimate as if an average level developer is doing the work.
8. Document your chunks of work in a spreadsheet
Some PMs may feel the allure of putting things into their PM-tool (Jira, Asana or the likes) but I recommend sticking to spreadsheet in this phase still. This is to derive a budget, not to create a backlog for a project. The spreadsheet allows you to move rows around, adjust daily rates, dropping rows/features to see how certain parts affects your budget.
9. Estimate in work days
"Points" or "complexity" is just adding a layer of abstraction which won't help here since you're going to need to convert it into time. Don't do hours since it's far too granular and implies too great of an accuracy of the estimate. Keeping it in days makes it natural to bake in workday overhead in the estimates. That is spending time replying to emails, reviewing pull requests and discussing technical solutions. It's ok to put in e.g. half days or zero days if the task feels like a rounding error.
10. Do many independent estimates
If estimating together you risk group conformity taking precendence over independent thinking
Ideally three independant estimates. Use your first spreadsheet tab as a template, then make a copy for each estimator. Ask them to estimate without looking at eachother's estimates. If estimating together you risk group conformity taking precendence over independent thinking, or a strong personality overriding everyone else.
11. Do three-point estimations
Estimators are more comfortable giving ranges. It'll also be more accurate and large spreads will reveal uncertainties. For each task ask them to produce a Optimistic (minimal), Nominal (Likely) and a Pessimistic (Maximum) estimate in your spreadsheet.
Create a calculated column with an "Expected estimate" with the formula
Expected = (Optimistic + 4 * Nominal + Pessimistic) / 6
In addition to this you can calculate the standard deviation. Tasks with a higher than normal standard devitions (outliers) are likely to contain uncertainty.
Standard deviation = (Pessimistic - Optimistic) / 6
I like to stress to estimators that they'e allowed to be really pessimistic in their pessimistic estimates if they suspect danger. Legacy or buggy code, unweildy dependencies and other unforseen problems always crop up in some part of the work, and when they do, it can take multiple times longer than the nominal estimate.
There'll be a template spreadsheet at the end of the post.
12. Before estimating, review the spreadsheet and chunks of work together with estimators
There should be discussion and adjustment of your chunks of work as estimators ask questions and assumptions are made. You may also split the work further or add new tasks. Write down any clarifications in the descriptions.
13. Be conservative in estimates when it comes to new tools
This includes new techniques, libraries, frameworks, architectures etc. People can be rather good at estimating when they're e.g. using familiar software architectures. Anything new is risky. If it can't be avoided then this should be reflected in the estimates. People tend to have optimistic views of new tools they read about online, but the picture is often not as rosy as they imagined once they start using them.
14. Be conservative in estimates when it comes to new integrations
If it's a well known public API it's usually easier to predict, but it can be risky integrating to internal services owned by other teams. They may not have an api, and/or they may not be incentivised to help your project. Fully booked calendars, long meetings, delays, are the norm in some organisations.
15. Quality is more work than we expect
There's a huge difference between a prototype/walking skeleton/MVP and something really smooth and jaw dropping. Sometimes you want to get something quickly out, but it might be slightly crude and untested. Sometimes it's the headline feature which has to be really polished with perfect test coverage. Make sure estimators know which level of fit & finish we're talking about.
16. Take into account work day overhead
If you're a developer, every day you'll be discussing technical solutions, reviewing PRs, answering emails, meeting ad-hoc etc. Some reports say that only 55% of the day is spent actively coding. Yet all of that other work is also work. Estimators should take into account they'll be doing all those other activitates during their workday, even if they're not explictly mentioned anywhere.
17. Review estimates together, then discuss and re-estimate until you reach consensus.
If estimates from different estimators are far off from eachother, they usually had different conceptions of some of the work chunks. You should discuss to discover from where the discrepancies come from and calibrate the estimates.
Your numbers will be questioned, and you should be able to explain them.
It's too expensive
It's not productive to discuss an estimate as a whole. Stay objective, break down the estimates for stakeholders and find particular tasks which raise concerns. You've had multiple experts weigh in on each task, so you are well capable of giving good explanations for why an estimate was arrived at. Usually this is all that's needed to gain trust and confidence that they reflect reality.
Finding compromises
Sometimes the estimate can't arrive under a certain upper budget limit. Acknowledge stakeholders' concerns and offer them options.
Certain levels of quality increase spent time. Tradeoffs can be made with mutual agreement, but these should be clearly stated and accepted.
Can certain features be dropped?
Can the project be phased further, delaying certain features?
Give stakeholders control to adjust the scope.
Don't under-estimate to "win"
If expectations aren't aligned with reality and good enough compromises can't be made, then it's better to call off a project before it starts. Otherwise you're looking at a list of the following consequences: missed deadlines, money problems, broken trust, burn outs, low morale, poor quality, failed launch, loss of business and reputation.
Everyone loses. A good estimate should aim at project success.
"Parkinon's Law" states that "work expands so as to fill the time available for its completion". That means even if you overestimate a task, the one working on the task is likely to use up all the time as estimated for the task even if they could be done faster.
If this is a point of worry then don't bring the estimates along from the spreadsheet to the backlog in your PM-tool. Some tasks will be done faster, some slower. The only thing that matters is the total expected sum.
If the total project work duration is under the expected estimate, then it can be used for extra development, increasing polish and quality overall, fulfilling Parkinson's Law, but budgets tend to become self-fulfilling prophecies.
Accurate scheduling might not matter to you, but if you want to use the estimates to produce a schedule then consider these things.
Team size matters
Adding more people won't deliver a baby any faster
There are very steep diminishing returns after each new person you add to a project. The second developer might double your speed, but the third might only give a 20% boost. It all depends on the project. In many cases adding more people won't deliver a baby any faster.
No one is present all the time
Sick-days, vacations, training days etc are likely to affect your timeline. Assume someone is only present 80% of the time if setting a deadline in some point in the future is important for you.
On-boarding bears real costs
People are not interchangeable. If someone is switched out to a similar profile, there will regardless be a cost in terms of immersion time. If you're dealing with an agency, you can agree with them to bear the cost of any such changes.
