In 2008, I started a glass company without any construction office experience. Of course, I came to regret that later, but it allowed me the freedom to work out processes without a bias toward the way things should be done.
I have no formal training with respect to construction estimating, but if I had been trained on a fool-proof system, we wouldn't have invented BidUnity. Without training, I have always been free to recognize the insufficiency of my firm's processes. After years of research and efforts to implement best practices at my own firm, I have come to understand why productivity within the construction industry has not improved on par with that of other industries. The reasons are many and complex, but understanding how things will be different in the future requires understanding how they became what they are today. We can then explore how new technologies are being applied to unsolved problems and why solving those problems will disrupt the construction industry.
There are four groups driving the construction industry. No project can be completed without their mutual collaboration; however, the business processes which have been adopted stifle the natural progression of the industry toward greater efficiency. Based on the technologies available to the industry, the participants have adopted a fair approach to collaboration by any standard. The good faith expressed through the way the parties presently interact provides a strong indication that new technologies which reduce risk and promote fairness will gain widespread adoption. The following is a simplification of the groups and their constraints in participation.
After developing their plans for a project, owners have only to solicit and compare cost estimates from several qualified, competing firms. There are differences in the qualification processes between firms, but owners have the unique opportunity to establish their costs of construction.
General Contractors limit their investment in the process of researching construction costs by farming out the details to sub-contractors. The primary risk that general contractors must manage is in making sure that they have accounted for all required work in their proposals. Secondary to that, they need to qualify which of the received proposals should be used to establish the values of each required type of work.
Material Suppliers face less uncertainty than other participants with respect to individual construction projects. Material suppliers provide trade contractors with price quotes which are valid for a specified time. Some material suppliers reduce the risks of specialty contractors by determining parts lists on their behalf, most do not. In general, material suppliers offering complex building systems will provide this service. In most cases material suppliers dictate the terms of the sale to trade contractors.
Trade contractors are best positioned to determine specific costs for the types and quantities of work because they are responsible for its performance. Best practices dictate that they maintain detailed cost records for the performance of similar projects and systems which can be used to accurately predict costs of the construction of new forms which use similar building systems. In actual practice, trade contractors apply their experience and specialized knowledge to determine labor, material, and equipment requirements which constitute their primary costs of performance. Due to the complexity of the information which trade contractors manage, computer based systems have not been able to better address the needs of specialty contractors than experienced tradesmen; however, machine learning platforms will change this in the near future.
To create a proposal, each firm must identify and make assumptions about the following:
The owner specifies what, where, and when to build.
Material suppliers have very little risk with respect to their proposals. They specify a price for a quantity of material. Most of the time they are not responsible for the determination of the quantity specified.
Construction management firms typically specify what, how much, and how long on their proposals.
The most difficult risk for construction managers to control relates to how long the project will take. Because they are dependent on subcontractors to complete the work on schedule, they include language in their contracts to transfer risks of performance with respect to time to subcontractors. In practice, it is difficult for them to fully transfer these risks with respect to minor delays by subcontractors.
Recall that the industry expects trade contractors to maintain detailed records which can accurately predict costs. In practice, the proposals of trade contractors are highly subjective. Construction managers can attest to wide spread variation which sometimes makes determination of an accurate market price difficult. Their historical budgeted versus recorded costs contain sufficient variance in what and how much such that additional stipulations are not internally meaningful. Trade contractor proposals are evidence of the insufficiency of their systems. The construction schedule has a material effect on specialty contractor's costs because they supply labor services at regular time or time-and-a-half, yet the majority of specialty contractor proposals do not include stipulations as to the schedule of performance.
The variance in recorded data, particularly the data of trade contractors, is the largest single source of risk in construction today. Trade contractors which continue to be successful within the industry have adopted unique approaches to managing these risks, but those approaches primarily pass risk to sub-subcontractors which does not reduce any actual risk. Instead, transferring risk to another tier introduces other types of risk to the project.
Every industry comes to recognize standards as many participating firms solve problems in similar ways. The key to construction management firms' success is managing risks well. By accepting risks, they gain opportunity to make profit. They assure profit by divesting all of that risk to other firms without regard to whether those risks become better managed.
The commitments construction managers make as to the what component of their proposals to the owner contain risk with respect to the quantity, type, and value. They pass this risk on through subcontract by requiring subcontractors to furnish all of the what as shown on the owner's plans and as specified therein. Only the value of the trade contractor's proposal is used in the contract. Their proposals are specifically excluded from the contract documents because reliance upon the trade contractor's specifications of quantity and type introduces risk to the construction manager. This is to say that if the trade contractor's estimating staff made a mistake and did not account for some portion of the work, the trade contractor would still be required to provide that work at no additional cost according to the terms of their contract with the construction manager.
Construction managers are responsible for the schedule. Their costs are directly related to the project's duration as they budget for supervision, storage facilities, temporary offices, and more based on the expected duration of the project. When project schedules run over, their budgets for these items fall short. In addition, they may have accepted responsibility for completion based on a certain date in their contract with the owner.
To reduce risks associated with the schedule, it is accepted practice to require subcontractors to perform work on the project's schedule which is specified by the construction manager. It is often further agreed that the construction manager has the right to revise the schedule from time to time as they see fit. This becomes a material provision of the contract when it is agreed that time is of the essence. This contractual language establishes the right of the construction manager to seek damages from trade contractors which contribute to delaying the project's schedule.
In addition to establishing the time for completion, construction managers further offset schedule related risks by requiring that trade contractors perform their work on schedule at no additional cost. This provision, if agreed upon, limits the trade contractor from seeking additional funds to offset increased costs of performance resulting from overtime, diminishing returns from labor as more workmen might be required to meet the schedule, and other costs which could change based on scheduling constraints.
The overwhelming majority of construction management firms do not have the assets, liquid or otherwise, to build the projects under their management. After all, the owner is expected to pay for construction. Because construction managers do not have the funds and because there is a time-value associated with money, an obligation to pay for completed work prior to receipt of payment by the construction manager presents another form of risk to construction managers.
Construction managers address this risk through pay-when-paid clauses. Their contracts stipulate that payment is not due to subcontractors unless and until payment for the subcontractor's work has been received.
Most projects use bank funding through a construction loan. Construction financing practices vary by location, but are most heavily influenced by state law. States establish the legal framework through which contractors can assure payment for their participation in construction projects. Banks observe and follow the guidelines established through state laws.
Banks' observance of state's laws with respect to paying for completed construction work often have the unintended consequence of delaying funding for projects. Most states recognize a difference between tangible personal property and real property. Specialty contractors are the firms which convert tangible property into real property through its combination with labor and subsequent incorporation into the building structure. Banks which are providing construction loans are providing those loans based on the value of the real property which is derived from completed work in place.
Trade contractors encounter a unique risk with respect to the timing of receipt of payment in combination with delays to the schedule. Materials having long lead times need to be ordered in advance. The risk is that materials are ordered and received, but that the project is delayed. There are ways to secure payment when events take this form, but the risk is that there are costs associated with pursuing those options without the opportunity to recover those costs.
In addition to recovering costs associated with materials when their installation is delayed, trade contractors also incur costs when project duration is extended or their work is delayed by others. To limit disputes, construction managers typically require trade contractors to agree that the only recourse for delays of the performance of their work is an extension, equal in time, of the time to complete their work.
Managing risks can be complex because it is so difficult to anticipate every possibility, however remote. Risk in construction is generally accepted by the party which has the most control over that specific risk. General contractors accept risk from the owner and distribute it through subcontracting that risk to other firms. In this way, the total risk associated with any given project is shared between many firms, primarily trade contractors.
Recall that qualification of firms and their proposals by owners and by construction managers takes place during the estimating process. A qualified proposal is a proposal which is at or about the market price. Qualified proposals represent a fair value for the work to be performed. The market price can only be ascertained by soliciting several proposals for comparison.
Ironically, trade contractors are at a unique disadvantage when it comes to determination of market prices. The industry offers trade contractors very few opportunities to review the competitive bids of other specialty contractors, especially proposals for the same project. Price discussion among competing construction management firms is widely accepted and even the standard for public works projects where sealed bids are required to be publicly unsealed. As result, many trade contractors have adopted rule-of-thumb type approaches to predict market prices which they may or may not be able to incorporate into their bidding processes.
Qualifying firms and proposals is different. Qualified firms have sufficient liquid assets to complete the work and they will agree to perform the work for the proposed value. For a construction management firm to fully divest the risks associated with performance, they must be willing to pay the market price for the work or they must contractually obligate a qualified firm to perform. Liquidity is an important consideration because damages from a party with no financial assets cannot be recovered. Both approaches, paying a fair price or contracting with a qualified firm, are effective in transferring the risk from the construction manager to another party.
The latter approach to risk management by construction managers sets up an obvious conflict of interest between project participants. Recall that construction managers often provide a fixed price to owners. It should be assumed that they construct their proposals based on market prices, but whenever a qualified trade contractor provides pricing below the market price, owners and/or construction managers have an opportunity to make additional profit by paying less than the market price.
When trade contractors provide proposals, they do so based on certain assumptions about how the work will be performed. For example, a glass contractor providing glass storefronts is not likely to consider that the construction manager will schedule the concrete for sidewalks to be poured on the same day that glass is scheduled for delivery and installation. In that situation, the productivity of both trades will be reduced, but through contract neither trade may have recourse.
Machine learning offers the promise of being able to precisely quantify the cost of scheduling choices which impact performance of the work. As such tools become available to trade contractors they will be able to evaluate costs more objectively and will become less willing to accept the scheduling risks in the same ways that they do today.
Trade contractors receive funds from construction managers. They understand that funds can be delayed for many different reasons, but they generally have no way to verify when the construction manager receives funds or if the construction manager unreasonably delayed payment. They can keep track of how long each of their customers takes to pay invoices and price the work of those customers accordingly. This is not a practical approach, however, because of the limitations of trade contractors to implement best practices.
Smaller firms have fewer resources to fix problems, fewer error checking processes, and as result they inherently carry greater risks than more established firms. The complexity of trade contractors' businesses has been and continue to be the drag on construction productivity. Their businesses are cumbersome enterprises. Not only do they source more than 80% of construction materials, but they also perform the complex tasks of assembling structures. Where gains in productivity have been made, they result from better planning of the upper tiers and the ability of management firms to engage a greater number of relationships with specialty contractors; however, the industry will always be governed by the complexity of the trades.
It is of great importance to recognize that the project participants which accept the most cost risk are at the greatest disadvantage with respect to ascertaining the market price prior to contracting. Because this is a grave injustice, tools which can remedy this condition are ripe for wide-spread adoption and acceptance by the industry.
Trade contractors will be the primary beneficiaries of machine learning within the construction industry because they bear the majority of cost risks. While they exert far more control over construction costs than other participants, the contract terms which they often assent to include risks which are often beyond their ability to control. Their assent to the assumed risks is generally regarded as being fair because they agreed to accept them; however, when the risks are not fair, meaning that the party accepting the risk cannot directly control it, distribution is justified when the perception of the potential for costs associated with the risk to arise is small or the total value of the associated costs are expected to be small in relation to the contract value.
The acceptance by trade contractors of risks which are beyond their control has crippled their ability to progress. The additional costs which they incur stifles their ability to accumulate the resources to scale their businesses. The limited ability of these firms to profit is evidenced by the complete absence of national players engaged in work of all sizes.
Ultimately, the market is to blame. The case can be made that we have an efficiently functioning market with tight margins. Given the lack of productivity increases within the industry and high costs of construction, a case could also be made that we do not have an efficiently functioning industry.
Recall that trade contractors can participate in construction projects when they can provide the best value in consideration of the total cost and time to complete. Recall their unique disadvantage with respect to determining the market price. Recall the limitations of systems used by trade contractors in abstracting previous performance costs to new, different projects. Also recall that determination of the market price can be unclear even in consideration of proposals from several firms.
We have a functioning market where people are establishing prices based on their experiences. This is not unlike any other market in history, except that this is today. As soon as better tools become available, they get used. To stay competitive, everyone will adopt the best approach available to them. This is why so many construction estimators use spreadsheets and only a few use paper and pencil.
For specialty contractors to become involved in projects, they have to provide the best value. This is essentially the same thing as charging the lowest price for the work because, recall that they do not control the schedule.
Imagine if there were a firm which could price each project perfectly. Assume that pricing projects perfectly meant that the firm charged a reasonable price for each project which resulted in the stable recognition of a margin which is equal to a reasonable business profit.
Imagine being a trade contractor engaged in competitively bidding against this firm. If your number is higher, you do not get the work. If your number is lower, you do not realize a reasonable business profit. This dynamic is what makes machine learning particularly disruptive with respect to the construction industry. There are people who still use a paper and pencil in lieu of spreadsheets, but there will not be trade contractors who still guess at the market price--they will never realize a profit in competition with firms that utilize better tools.
As various trade contracting firms fold, talent will aggregate at the remaining firms which leverage machine learning technologies to provide better estimates of construction costs. As these firms get better people, they will grow in size. Their increased size will allow them better economies of scale and subsequently additional resources. Those firms will be in the best position to take advantage of automation technologies, but that is beyond the scope of this discussion.
Our previous discussion explained why the construction industry will readily accept machine learning tools. While adoption of the tools will reduce risks and provide market participants with stable outcomes, we also provided reasoning to suggest that other market participants will not have a choice.
To understand how the technology will be applied within the industry we need to understand how it works and then we can best determine where it should be applied.
First, we must remember that the construction process is inherently complex. It involves many different building systems being assembled together in the correct sequence and in the correct manner. While many building systems are very simple, their assembly and installation is not. As evidence of this complexity, consider how many national franchises are offering the assembly and installation of construction materials.
Second, consider the unique, pattern-breaking nature of projects. Most large businesses started small, repeated the same processes while striving to improve efficiencies, and grew into large companies. To underscore the unique nature of projects, consider the greatest number of identical buildings that you have ever seen on the same street. Patterns which have supported consolidation and productivity increases in other industries have an abstract form within the construction industry.
The net result of variation is that most, more probably all, trade contracting businesses have not been able to fit their data into systems in ways that lend themselves to traditional statistical approaches. That is not to say that traditional approaches to analyzing trade contractor data are a total failure, but more specifically, analyzing production rates often results in sufficiently high variance such that human prediction is often still better.
The reason that human prediction has continued to be better for so long is because humans are able to understand how the large number of variables associated with each project have influenced results whereas data structures just have not been flexible enough to support machines in making those distinctions.
Advances in data gathering have unsurprisingly been followed by greater insights into the relationships between variables. Recently it has been reported that Watson, billed as IBM's cognitive supercomputer, is being used to help doctors with patient diagnosis. As it turns out, Watson is doing an excellent job.
Data is the key. In the case of the construction industry it is the key to unlocking the potential of computers to better predict outcomes than people. Unfortunately, the complexity of the trades has delayed the development of tools required to log data in ways which lend themselves to data analytics. Other factors have contributed to the delay, in particular it is not in the other participant's financial interests to clarify the costs of the risks relegated to subcontractors.
Recall that the majority of trade contractors are operating small companies. Small companies are data silos. Even though my glass company was able to categorize data in meaningful ways, we observed so much variance in regressions that we were unable to design useful tools for improving our estimates beyond using our own judgment for constructing proposals. Besides limitations imposed by small data sets, the introspection of a single firm is of little value when seeking to determine the market price.
It can take a lifetime to master a trade. Learning a trade is less fashionable than it used to be, but in many ways, this is because the trades have become more complex. There are more acceptable methods of building, more specializations, new and different materials, alternate products, and of course the way that things used to be done. When you think about it, and every construction business owner has been thinking for a long time, it is difficult to figure out how to categorize construction information in meaningful ways.
As a result, most construction firms organize project information by project. They are content with maintaining data across many different systems because they do not lose insights which otherwise would be unavailable to them if they had their information in one place.
Because the collection and meaningful organization of information is a prerequisite to machines effectively helping us, we need a new way to store construction information. NoSQL database systems offer the promise of being able to store massive amounts of data and of equal importance, they can store data in a more flexible manner than traditional SQL data bases. SQL stands for structured query language--as we all know construction information is difficult to impose a regular structure upon. BidUnity is the first machine learning platform available for use by specialty contractors and backed by a database designed for big-data.
Many firms treat the estimating process with less regard than it deserves, but by implementing a best-practices approach to estimating, we can leverage the process to structure data. The estimating process is used to form budgets for performance, where assumptions can be documented, and if we are able to impose sufficient structure at this stage of the construction process, we have the ability to determine how subsequent data records should be related to the project.
Everyone knows that estimators have plenty to do already, especially estimators. Actually, when we started down the path of developing BidUnity our goal was limited to providing the industry with a better way to approach estimating. We developed a much faster, more consistent approach to estimating with numerous other features that are beyond the scope of this discussion. We understand that a better tool for estimators is a prerequisite to organizing construction information.
Features which are relevant to solving the problem at hand and which are included in BidUnity's programming allow trade contractors to configure complex product systems in various ways. It supports structural analysis of constraints, recognizes equal alternate products, equivalent product systems, calculates required materials in units, lengths, areas, and volumes. Perhaps best of all, BidUnity's APIs allow us to communicate with other computer systems including those of our users' material suppliers. Because we understand the value of time, we provide users with the ability to save theirs. We understand that some people are still going to prefer spreadsheets and not everyone likes new technology.
We aim to answer the question of how trade contractors will teach machines with BidUnity. An exploration of the construction estimating process is required to further understand why BidUnity promises to lend structure to the industry's data.
Both complexity and confusion can be introduced into the processes of estimating; however, the exercise as well as its objectives remain constant. Firms strive to predict the costs of building as accurately as possible. Various methods may be employed to create estimates; the selected method will dictate how to conduct each of the processes. It is important to recognize that different approaches to estimating are employed based on combinations of three factors.
To address each case, the estimator might choose to go about preparing a proposal differently. Early on in the project, in the design phase, estimators may choose to provide some basic cost information, but not complete a proposal. Further along in the design stage, estimators might have enough information to work up a proposal, but this is still a little early for them to invest a significant amount of time. When the amount and quality of predictive information is poor it requires much more time to create a proposal. We hope that the flexibility of BidUnity can assist with projects of this type, but it needs a lot more information before it can be of real use in these situations. The estimator's limited available time is often the most influential determinant.
We anticipate that BidUnity will help companies adopt a more consistent approach to estimating because it saves estimators so much time. BidUnity is good at steps three, four, and provides the structure to predetermine step six. Not only does it do half the work, but it also relies on the estimator's experience and judgment to customize the proposal according to the needs of the project. To save estimators more time, it writes proposals, product submission documents, and even more. Again, we understand that saving time by providing a more convenient platform is required in order to get estimators to try this technology.
While the estimating exercise is simple, in my early days the variation among projects and types of work obscured my ability to recognize a consistent approach. As my company grew I hired people that had backgrounds in estimating and they brought their experience to my company. I learned a lot from them. I hired excellent people, but I also observed that their approaches suffered from the same inconsistencies. Their individual approaches were easy for me to accept because they were so successful, but finding great people and letting them do their thing eventually caught up with us.
I researched ways to improve my estimating strategies for several years. That research included reading, but I quickly discovered that written resources on this topic are redundant--they describe the process above which is well established and simple.
I asked my business contacts about their strategies and quickly realized that most people used spreadsheets and that their processes were equally as subjective as my firm's. I didn't formalize my study, but at least 9 of 10 people admitted to me that their approaches were always based on educated guesses not unlike my own. While nearly everyone used spreadsheets to organize cost information, they also rely on multiple other programs. They use industry specific programs to determine quantities of parts to order, request quotes from suppliers for those materials, and assemble their cost information in the spreadsheets. Some use the spreadsheet to format their proposals, most use another program for this purpose.
My friends outside of the construction industry don't understand why subjective approaches are so prevalent, but they also don't understand the endless variation among projects. Construction estimating will always be a subjective approach to realize the most fair values in consideration of many subjective variables. Anyone can create a system for bidding construction work by using a spreadsheet and everyone is aware that the process is not perfect.
My research indicated that no firm has yet removed subjectivity from its estimating process, but most have adopted practices which promote consistency and approximate the market price. The widely accepted standards of practice are only the direct result of the limitations of available tools, the unique nature of construction projects, the extraordinary complexity of information, the knowledge required to thoroughly master any single construction discipline, and the limited resources of most construction firms.
After accepting that a subjective approach might be required by my firm's estimating process, I continued to research best practices to realize process improvements.
Construction firms strive to reduce uncertainty in cost estimates; the quality of an estimate can be judged by the amount of uncertainty included in its constituent items. Best practices serve us to reduce the inherent uncertainty.
This information discussed how estimating is practiced across different construction firms. It explained methods of construction estimating, explored the limitations of traditional systems, and discussed best practices for improving the quality of traditional approaches to construction estimating. It explored the obstacles which firms encounter when attempting to implement best practices. It explained how the industry will overcome the obstacles to implementing best practices and why the wide-spread adoption of best practices will fundamentally disrupt the industry. It also explained how machine learning will be used to enable firms to follow best practices.
Humans will always be required to guide markets and this is especially true of the role that estimators play in the building construction market. The point of applying machine learning technology to the construction field is not going to be the replacement of people, but rather to promote the learning which will enable the industry to distribute risks more fairly. In consideration of how the industry presently operates, given the available tools, it is hard to imagine a fairer approach to doing business. Of course, there will always be opportunities for firms to take advantage of less sophisticated firms, but it seems very likely that in the near future trade contractors will have better available tools for learning, sharing, and for rebutting a portion of the risks unjustly imposed upon them.
How will trade contractors use the information that they are able to unlock by applying machine learning technologies to their businesses? In addition to improving the efficiency of their businesses, it will be used to enhance their right to payment for risks beyond their control. Most likely, it will result in shifting at least a portion of the construction cost risk back to construction managers and owners.
In addition to lowering their risk, we also expect to be able to provide trade contractors with bench marking information. We plan to answer how well their firm executes similar processes relative to other related firms. We will answer how much their firm spends on items relative to other similar firms.
Perhaps most importantly, we expect to be able to help firms identify their good people and to provide concrete information with respect to how much they contribute to the firm's productivity. We have developed algorithms which can determine the level of productivity of individual workers with respect to the average of that firm's total workers.
We look forward to applying these technologies to your businesses and assisting in the implementation of best practices at your firm.