The change detection term evaluates how often the proposal correctly picks up on the construction of new buildings. In other words, it measures the model’s ability to characterize what stays the same as time goes by. The tracking term evaluates how often the proposal correctly tracks the same buildings from month to month with consistent ID numbers. The SpaceNet Change and Object Tracking (SCOT) metric combines two terms: a tracking term and a change detection term. In fact, all existing metrics that we investigated unfortunately proved to be a poor fit for our dataset and challenge task, leading us to develop our own metric. Since we are tackling a very hard problem in SpaceNet 7, MOTA is therefore a poor choice. The end result is that for difficult scenes with few true positives and many false positives, it is possible to achieve a negative MOTA score. Yet this metric is not a good choice for difficult sequences, since errors are compounded additively (rather than being “averaged” out via a harmonic mean with the historical SpaceNet metric). We originally intended to use the Multiple Object Tracking Accuracy (MOTA) metric for SpaceNet 7, which is a commonly used metric for object tracking in video. Measuring the concordance between the ground truth and proposals is the job for an evaluation metric. Each footprint is assigned an ID number, with the idea being that each reappearance of the same ID in subsequent months corresponds to a new observation of the same building. The SCOT Metricįor SpaceNet 7, the ground truth and the model-generated proposals both consist of a set of building footprints for each month. This blog details the advantages and specifics of this new metric that will be used to score the impending SpaceNet 7 challenge. These two competing priorities (tracking and change) are not fully captured in existing metrics, and lead us to develop a new metric: the SpaceNet Change and Object Tracking (SCOT) metric. Ideally, we would like to quantify the ability of machine learning algorithms to correctly predict when and where the change takes place. Secondly, there is significant construction activity in the SpaceNet 7 data cubes, with new buildings appearing (or disappearing) throughout the time series. address), which we would like to track over time. For one, each building footprint is assigned a unique identifier (i.e. How exactly one should measure performance for the SpaceNet 7 task is a tricky question given the multiple dimensions of the dataset. Furthermore, the unique SpaceNet 7 dataset poses a challenge from a computer vision standpoint because of the small pixel area of each object, the high object density within images, and the dramatic image-to-image difference compared to frame-to-frame variation in video object tracking. As detailed in our announcement blog, the goal of SpaceNet 7 is relevant to numerous human development and disaster response applications. The SpaceNet 7 Multi-Temporal Urban Development Challenge has the ambitious goal of tracking precise building addresses and urban change from satellite imagery. SpaceNet is solely managed by co-founder, In-Q-Tel CosmiQ Works, in collaboration with co-founder and co-chair, Maxar Technologies, and the other partners: Amazon Web Services (AWS), Capella Space, Topcoder, Institute of Electrical and Electronics Engineers (IEEE) Geoscience and Remote Sensing Society (GRSS), the National Geospatial-Intelligence Agency (NGA) and Planet. building footprint and road network detection). Preface: SpaceNet LLC is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e.
0 kommentar(er)
