As we’ve been working with more design teams, one of the concerns that occasionally comes up is whether design engineers really want to share all of the data they are generating. Getting a design closed requires constant experimentation. Sometimes the experiments work, other times things go way off the tracks. If the data is simply collected, but is not viewable or filtered by engineers, managers may make decisions off some experimental result could create some uncomfortable situations.

One of our goals from the beginning of designing Pinpoint was to make the data useful for the engineers to use. Presenting the information both as a high level summary as well as allowing the engineer to dig into the details of what is happening on a various experiment. We’ve always provided engineers with the ability to label experimental runs, add comments to provide context, and hide snapshots that represent bogus data. In addition, we added the ability for engineers to mark certain snapshots as representing reviewed data and summarize this up to management as a status report.  All this to prevent people using the data for conclusions that may be inaccurate.

In Pinpoint 2.5, we added the ability for engineers to automatically mark snapshots as private. Even though we’ve seen that open communication among a team transforms communication, we wanted to make sure that the tool could be useful even when running wacky experiments that an engineer wants to keep private.This allows them to still check the data into Pinpoint and see the summary of the results or compare various experiments but not worry about any failed experiments being seen by others. Once the engineer decides that they would like to share the information, they can click the publish button and then share the snapshot with others.

Our goal continues to be helping to improve engineers lives and helping make teams as productive as possible. Helping to provide flexibility on managing the data is one of the ways we do this.

When we see data presented, we start looking for reasons that the data is that way. We want to know why. Why is the stock market up? We want to know and if no one helps us along that path, we’ll make up a reason. When teams start collecting data on their projects, the raw data is looked at by everyone, but only the people who are closest to the data know what the data means. We realized that when teams are working and management is looking at the current metrics of the project that there is a need to mark certain measurements as not only reviewed but also add comments.

This is one of the reasons we built the ability to add “best so far” flags into Pinpoint. This allows an engineer to flag a particular snapshot of the metrics as the best run so far as well as explain why things are the way they are. This flags a run so that management is only looking at runs that have been reviewed and are immediately provided with the associated narration that goes along with the data. Moreover, the running narration over the course of doing a design provides the story of an entire block through the design flow. By the end of a project, it’s difficult for everyone to remember what was going on 2 months ago, but being able to look back and remember can provide insights into how to do better on the next project.

How are you currently managing the narration that surrounds the metrics for your design?

When we were in the middle of beta testing Pinpoint, I had a conversation that enlightened me about the complexity of creating a physical design dashboard. At one of our beta customers sites, I had checked in a few snapshots of the data for the most recent runs I could find for each of the blocks. Not ideal for actual usage, but important to help show the kinds of visualization Pinpoint offers.

I was giving a demo to the PD manager Vishy* and his boss Scott*. After showing some of the pieces, Scott said to Vishy,

“So this is the real data then. I should be able to look at this and tell whether or not we are almost done. This seems a little different from the status you were giving at our last meeting.”

Vishy responded, “Well, I’m not sure what data is checked in here. Some of this is probably old.”

Scott concluded, “Okay, so this tool is really only for your group’s use. I shouldn’t make any decisions based on what I’m seeing here.”

This is of course antithetical to Pinpoint’s purpose, but Scott’s point illuminates a critical problem:

In order for visualization to be valuable to management, the data must be clean and current.

Basic housecleaning can be pretty simple. Engineers need to prune failed experiments, correct specific metrics when a datum is flawed, and mark their “best so far” runs so that status reports show their actual progress, and not all the clutter of experiments. Adding a few notes to help put the numbers in context can also add a lot of value. Scrubbing data at this fundamental level gives management confidence that they are looking at real information they can use to make decisions. That leads us to the second critical problem.

The only way to ensure data is clean is to give engineers a good reason to clean it, regularly.

Most people don’t love to clean house for its own sake, so the scrubbing data has to be in the service of performing an engineer’s day-to-day job. This is where so many homegrown solutions fail. They automatically collect information but don’t provide tools that are useful for the engineering team. As a result, the data is a hodgepodge of random experiments, runs, and iterations — only slightly useful to the engineers and hardly ever useful to management.

The challenge we are eagerly tackling:

Make the data valuable by giving engineers compelling reasons to scrub it, all the time.

Closing chips is hard and it’s getting harder. As we discuss Pinpoint with people, we find that organizational memory is one of the pervasive problems in the industry.

We all have a limit for the amount of things we can keep in our head at one time. We remember the mistakes that we made most recently, but tend to forget the ones that haven’t bitten us in a while. This can cause problems when you close chips, managing ever increasing complexity, hoping you’ll remember all the various gotcha’s.

The more complex the chip, the more each engineer must specialize, creating unique solutions to unique issues. Passing this learning on to the team is very difficult.

Complexity also brings unexpected side effects. Just this last week, we heard a story of a team who spent a good deal of time putting in place clock gating for the whole chip. Things looked great until they pulled in the logic from the BIST module which introduced an extra 20,000 ungated flops into the design. It was a power nightmare.

Key problems facing teams go beyond the difficulty of tracking metrics. They must also make sure the entire team can see the effect of each member’s changes on the chip at large as well as provide a mechanism to capture organizational learning so that past mistakes are not repeated.