[1906.01618] Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

The advent of deep learning has given rise to neural scene representations - learned mathematical models of a 3D environment. However, many of these representations do not explicitly reason about geometry and thus do not account for the underlying 3D structure of the scene. In contrast, geometric deep learning has explored 3D-structure-aware representations of scene geometry, but requires explicit 3D supervision. We propose Scene Representation Networks (SRNs), a continuous, 3D-structure-aware scene representation that encodes both geometry and appearance. SRNs represent scenes as continuous functions that map world coordinates to a feature representation of local scene properties. By formulating the image formation as a differentiable ray-marching algorithm, SRNs can be trained end-to-end from only 2D observations, without access to depth or geometry. This formulation naturally generalizes across scenes, learning powerful geometry and appearance priors in the process. We demonstrate t...

4 mentions: @quantombone@hillbig@hillbig@JoanIratxeta
Date: 2019/06/05 06:48

Referring Tweets

@quantombone Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations. The idea of differentiable ray-marching looks promising! https://t.co/qoksNZbH33 #computervision https://t.co/4g12Mr3pQc
@hillbig 複数の姿勢付き画像から、3次元構造を考慮した表現を獲得し生成する。物体クラス毎に位置から特徴表現を返す関数を学習し、画像生成時は視点からの光線と物体との交点をレイマーチングで解き交点から特徴表現を得る。GQNより正確かつ精細に生成 https://t.co/MlZG73Av3o https://t.co/iEG41V7t4q
@hillbig Scene representation networks give a structure-aware scene representation and can generate a scene from a new viewpoint. They use a map a position to a feature vector and render a scene with a learned ray-marching algorithm. https://t.co/MlZG73Av3o https://t.co/iEG41V7t4q