ICACT20230029 Slide.19
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
As future work, following topics need to be considered. Use of egocentric RGB in the proposed method is naïve. More suitable way of using the vision observation is needed for more human-like and accurate navigation. Reinforced learning may be implied. Furthermore, more closed receptacles need to be added to the active search. Addable receptacles can be Cabinet, Microwave, Drawer, Safe and Box. Finally, more effective methods of changing the list of subtasks needs to be discussed. Changing the list of subtasks could be implemented other ways than active search for more efficient adaptive decision making.
|
|
ICACT20230029 Slide.17
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In conclusion, We analyze FILM’s drawbacks and discuss how to improve the performance by modifying the deterministic policy.
These are the main contributions.
1. The use of visual observation reduces the reliance on the semantic map.
2. The agent can search for the target object by opening closed receptacles through active search.
3. We provide minor adjustment to overcome some semantic segmentation errors.
Therefore, We succeed in improving the performance by a little margin, showing that modifying deterministic method can improve the model without any learn-based method.
|
|
ICACT20230029 Slide.16
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Let us show a qualitative result. This example shows an episode with a high-level goal, “put a heated tomato slice on the table".
|
|
ICACT20230029 Slide.15
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Table shows the performance by task type. Note that among closed receptacles, only “Fridge” is added to active search in these results. Notably, task type “Cool & Place” improved with a significant performance increase (8.82%). It appears this most likely improved the overall result. The notable improvement on “Cool & Place” indicates that the active search functioned well, and the navigation of the goal object greatly affects the SR.
|
|
ICACT20230029 Slide.14
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
We evaluated the modified model on the ALFRED benchmark Table shows the modified method’s performance on valid seen split and valid unseen split against the original FILM. our approach performs slightly better than the original method both on Valid Seen and Unseen, showing higher SR by 0.74% and 0.36% each. On the other hand, GC is decreased by 1.42% on Valid Seen split and also decreased by 0.86% on Valid Unseen split.
|
|
ICACT20230029 Slide.13
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In addition, we added minor changes to avoid semantic segmentation error. First, we deleted the “sliced” object from the subtask template. The pretrained Mask-RCNN module struggles in distinguishing non-sliced object and sliced object. It both classifies both objects as non-sliced object, so it is meaningless to differentiate those two in the subtasks. Moreover, in the deterministic policy, there’s a part where the agent rotates the camera angle right and left repeatedly if it is arrived at the goal location but cannot observe any goal mask. The agent often trap itself in a camera rotation loop if it fails in semantic segmentation. This occasionally happens when the goal object is only observed partially in the visual observation frame. Thus, we modified the algorithm to move behind to get the whole picture if camera rotation loop happens.
|
|
ICACT20230029 Slide.12
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
To migrate the mentioned drawbacks, we propose the modified deterministic policy. With the modified deterministic policy, the agent uses visual observations for locating the target object and performs “active” search if the target object is not observed in the current environment by changing the list of subtasks. The figure shows the model overview. The background color denotes the original and the modified architecture of FILM. The part in green indicates the original structure, and pink indicates the modified structure. In the modified model, the Search Goal is selected as the coordinates of the target object in the semantic map or selected by the Semantic Search Policy for every 25 steps. But if the target object mask is observed in the Visual Observation, the Search Goal is changed to current location. Furthermore, the modified model also changes the list of subtask. If the target object is not found after enough steps, the list of subtasks is modified, and the agent opens receptacles for search.
|
|
ICACT20230029 Slide.11
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Another critical semantic segmentation error case is failing to detect the goal receptacle. In this case, the agent inevitably fails the task. For example, figure shows a failure case where the agent fails because it failed to detect dining table, which is the goal receptacle The agent cannot put the knife on the table even though the table is in front of the agent.
|
|
ICACT20230029 Slide.10
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Moreover, since the agent’s perception mostly comes from semantic segmentation, accurate semantic segmentation is important in ALFRED. However, completely avoiding semantic segmentation error is impossible. Thus, it is important to make an action algorithm which can overcome semantic segmentation error. One case of semantic segmentation error that occurs often is the detection of sliced objects. As you can see in the figure, the semantic segmentation model cannot distinguish between original object and sliced object.
|
|
ICACT20230029 Slide.09
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Furthermore, FILM does not open closed receptacles when exploring the environment. As you can see in the table, there are 6 kinds of closed receptacles in the ALFRED dataset and a lot of objects can be spawn inside closed receptacles. Therefore, the agent must look at the closed receptacles to reduce failure.
|
|
ICACT20230029 Slide.08
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Let’s look at the drawbacks more precisely. The first drawback is the heavy reliance on the semantic map. FILM only uses map data, for navigation. And the map data include semantic categories and coordinates.
However, Semantic map is often very noisy, especially when there are numerous receptacles in the provided scene. So, the agent is likely to struggle when locating the target object in this situation. Therefore, egocentric RGB, or direct visual observation needs to be used when navigating.
|
|
ICACT20230029 Slide.07
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Despite FILM is a significant model in the ALFRED challenge, its architecture poses some drawbacks. The navigation action relies too much on the semantic map. Thus, direct visual Observation needs to be used during navigation. Secondly, The agent does not make adaptive decision making, and the exploration is not “active” enough. For example, the agent does not open closed receptacles during the exploration process.
Therefore, the agent often fails the task if the target object is not immediately observable. Besides the architecture drawbacks, the performance of semantic segmentation is also essential since the semantic segmentation determines the agent’s visual perception.
|
|
ICACT20230029 Slide.06
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Also, to simplify the problem, FILM did not use low-level step-by-step instructions. Rather, it used map and search policy.
|
|
ICACT20230029 Slide.05
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The starter code of the ALFRED challenge, FILM used modular method. The figure shows the architecture of FILM. The modules include; Language Processing module, Semantic mapping module, semantic search policy module, and deterministic policy module. Note that all modules except the deterministic policy is learned. The Deterministic policy is the action module of FILM that decides the agent’s next action; Therefore, it is crucial to the model’s performance.
|
|
ICACT20230029 Slide.04
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
Early approaches of Embodied Instruction Following used end-to-end methods which included either concatenating language, visual features, and previous actions into one input or training a large end-to-end model. However, later works tackled the task with more modular methods, modularly processing raw language and visual inputs into structure form.
|
|
ICACT20230029 Slide.03
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
There are four evaluation metrics. SR, GC, and PLW version of these two. Success Rate only consider the result of the task success, which is defined as 1 if the object positions and state changes correspond correctly to the task goal-conditions at the end of the action sequence, and 0 otherwise. In contrast, the goal condition success is the ratio of goal conditions completed at the end of the episode. PLW SR and PLW GC are the Path-weighted version of both metrics that considers the length of the agent’s demonstration.
|
|
ICACT20230029 Slide.02
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
The ALFRED benchmark contains 7 task types; Pick & place, Stack & place, pick two & place, clean & place, heat & place, cool & place, and examine in light. Each task types are parameterized by 84 object classes in 120 scenes. The task takes place in simulated home environments, which is provided by the AI2-THOR simulator.
|
|
ICACT20230029 Slide.01
[Big slide for presentation]
|
Chrome Text-to-Speach Click!! |
 |
In the field of Embodied Instructions Following (EIF), the agent needs to understand instructions and execute sequential of subtasks involving interaction with objects and state changes of environment. Our work is done in the context of the ALFRED benchmark. ALFRED (The Action Learning from Realistic Environments and Directives) Benchmark is one of the embodied AI datasets that provides numerous language instruction data and interactive simulation environment, therefore allowing developers to test their agent’s ability to follow instruction in known and unknown environment In ALFRED, the agent needs to follow the natural language instruction using egocentric vision. The natural language instruction is given by the form of both high-level goal and low-level step-by-step instruction.
|
|
ICACT20230029 Slide.00
[Big slide for presentation]
[YouTube] |
Chrome Text-to-Speach Click!! |
 |
[Presentation] https://youtu.be/eMPVaF2cqPY
We introduce a modified module structure of FILM, which increases its performance on the ALFRED benchmark.
|
