Earlier this week I spent a lot of time debugging a random error that popped up and broke our CI/CD builds.
The Problem
We start our pipeline process by first running a build script locally before pushing our code out and making a pull request. The criteria for merging a PR into the master branch are:
- A build of the most recent code changes are successful
- A fellow engineer reviews and approves the PR
Being the awesome developer that I am, I, of course, skipped the part where I run my build script locally to catch issues prior to making the PR to begin with. So, of course, my feature branch build failed. Pro-tip: don’t skip running your local build script before making a PR.
Our build script does a lot of things, one of which is running msbuild. This part of the script ran correctly with no issues—but, in a downstream step—we copy files and move them around to generate a build artifact. The “copy files” step was failing because the main UI project’s obj folder ( we retrieve files from this folder to package into an artifact) was no longer there.
The changes made that broke the build were innocuous, so attempting to discover the root cause was extremely difficult.
The cause of the problem:
We are currently working on the modernization of a very old ASP.NET VB web application for our client. The new structure of the solution is based on the Clear Measure Onion DevOps Architecture.
The unit, integration, and acceptance test projects in the old VB project were not named the way we wanted. The projects were also using a testing framework we try to avoid. So, I replaced those projects with new C# ones. In doing so, the following chunk of code was removed from the solution file. This code was nested in with the old acceptance tests project declaration.
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "AcceptanceTests", "AcceptanceTests\AcceptanceTests.csproj", "{13F12812-AA0F-4A18-94AC-3A6DF3468D85}"
ProjectSection(ProjectDependencies) = postProject
{CD8E53D2-B177-494B-AE08-1CEEF98E43D7} = {CD8E53D2-B177-494B-AE08-1CEEF98E43D7}
EndProjectSection
EndProject
Removing the project removed these lines of code. I was not familiar with them so after snooping around a bit I came across this stack overflow page which explains their purpose.
Basically—from what I gleaned from the Stack Overflow answer—it’s a bit of a work around msbuild uses for creating interconnected project dependencies in your solution. These project dependencies have a direct effect on the project build order in the .sln file. These dependencies can be set and managed in Visual Studio.
I’m still a bit of a loss as to why removing the dependency caused msbuild to stop generating the obj folder for our UI project. The dependency really was pointless as it didn’t affect the acceptance tests in any way, nor the general order of the build from what I could tell.
It’s also unclear how or why the dependency was there to begin with.
The solution:
Ultimately the solution—which is I admit a bit of hack/work around for us—was to just re–add that chunk of code back into the solution file, nested under the new Acceptance.Tests project deceleration.
Hopefully this saves someone else some time or grief in debugging their own issue. There are still many variables to consider like:
- Does this issue have something to do specifically with the main UI (VB ASP.NET Web Application) project?
- Is it related to the mixing of VB and C# projects in the same solution?
- Is there truly a dependency the Acceptance.Tests project needs?
- Is it a bug in msbuild that’s just gone unnoticed because of the “edge case” nature of the issue?
Regardless, we got it working and we can now get past our broken build.