Khoa Vo

I am a PhD student at University of Arkansas in Fayetteville, advised by Asst. Prof. Ngan Le. I am working on problems at the intersection of Computer Vision x Deep Learning and I am also very intrigued by the problems of bridging vision and language understandings.

Before I joined UArk, I was a MSc student in the Department of Computer Science, Ho Chi Minh University of Science, Vietnam, advised by Assoc. Prof. Minh-Triet Tran. During this program, I luckily had an 6-month internship period under the advisory of Prof. Akihiro Sugimoto. My Master’s thesis is an attempt to resolve temporal action proposal problem in videos by a newly proposed method that learns adaptively attends to main actors.

Further in the past, I received my Bachelor degree in the Honors Program of Ho Chi Minh University of Science with the advisory of Prof. Minh-Triet Tran. In my final thesis, I studied the problem of image captioning and proposed a novel approach to improve captions quality over previous state-of-the-art. Thanks to the thesis, I made two publications in an international conference and a international journal, and achieved the highest prize in the Nation-wide Awards for Student Scientfic Research, Eureka.

News

Nov 28, 2023	One co-authored paper has been accepted to IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024.
Sep 15, 2023	I gave a talk at Southeast Symposium on Contemporary Engineering Topics, Arkansas Engineering Forum, held at Little Rock, Arkansas.
Jul 19, 2023	One co-authored paper has been accepted to International Conference on Image Processing (ICIP) 2023.
Apr 7, 2023	One first-authored paper has been accepted to The 5th Precognition Workshop at CVPR 2023.
Nov 19, 2022	One co-authored paper has been accepted to The 37th AAAI Conference on Artificial Intelligence.
Oct 2, 2022	One first-authored manuscript has been accepted to International Journal of Computer Vision.
Oct 2, 2022	One co-authored paper has been accepted to British Machine Vision Conference 2022.
Aug 17, 2022	I honorably received Graduate Fellowship for Spring 2023 from Rodger S. Kline Endowed Chair, CSCE Department, University of Arkansas.
Aug 17, 2022	I honorably received W.R. Thomas Endowed Graduate Fellowship for Fall 2022.
Jun 20, 2022	Our paper has been accepted to International Conference on Image Processing (ICIP) 2022.

Selected Publications

AAAI

VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioning

Kashu Yamazaki, Khoa Vo, Sang Truong, Bhiksha Raj, and Ngan Le

In AAAI Conference on Artificial Intelligence 2023
IJCV

AOE-Net: Entities Interactions Modeling with Adaptive Attention Mechanism for Temporal Action Proposals Generation

Khoa Vo, Sang Truong, Kashu Yamazaki, Bhiksha Raj, Minh-Triet Tran, and Ngan Le

International Journal of Computer Vision Jan 2023

Abs HTML Code

Temporal action proposal generation (TAPG) is a challenging task, which requires localizing action intervals in an untrimmed video. Intuitively, we as humans, perceive an action through the interactions between actors, relevant objects, and the surrounding environment. Despite the significant progress of TAPG, a vast majority of existing methods ignore the aforementioned principle of the human perceiving process by applying a backbone network into a given video as a black-box. In this paper, we propose to model these interactions with a multi-modal representation network, namely, Actors-Objects-Environment Interaction Network (AOE-Net). Our AOE-Net consists of two modules, i.e., perception-based multi-modal representation (PMR) and boundary-matching module (BMM). Additionally, we introduce adaptive attention mechanism (AAM) in PMR to focus only on main actors (or relevant objects) and model the relationships among them. PMR module represents each video snippet by a visual-linguistic feature, in which main actors and surrounding environment are represented by visual information, whereas relevant objects are depicted by linguistic features through an image-text model. BMM module processes the sequence of visual-linguistic features as its input and generates action proposals. Comprehensive experiments and extensive ablation studies on ActivityNet}}-}}1.3 and THUMOS-14 datasets show that our proposed AOE-Net outperforms previous state-of-the-art methods with remarkable performance and generalization for both TAPG and temporal action detection. To prove the robustness and effectiveness of AOE-Net, we further conduct an ablation study on egocentric videos, i.e. EPIC-KITCHENS 100 dataset. Our source code is publicly available at https://github.com/UARK-AICV/AOE-Net.
BMVC
Oral Session

AEI: Actors-Environment Interaction with Adaptive Attention for Temporal Action Proposals Generation

Khoa Vo, Hyekang Joo, Kashu Yamazaki, Sang Truong, Kris Kitani, Minh-Triet Tran, and Ngan Le

In Proceedings of the British Machine Vision Conference (BMVC) Jan 2021

Abs HTML PDF Code

Humans typically perceive the establishment of an action in a video through the interaction between an actor and the surrounding environment. Despite the great progress in temporal action proposal generation, most existing works ignore the above fact and leave their model learning to propose actions as a black-box. In this paper, we make an attempt to simulate that ability of human by proposing Actor Environment Interaction (AEI) network to learn video visual representation for temporal action proposals generation. AEI contains two modules i.e. perception-based visual representation (PVR) and boundary matching module (BMM). PVR represents each video snippet by taking human-human relations and humans-environment relations into consideration using the proposed adaptive attention mechanism. Then, the video representation is taken by BMM to generate action proposals. AEI is comprehensively evaluated in ActivityNet-1.3 and THUMOS-14 datasets, on temporal action proposal and detection tasks, with two boundary matching architectures (i.e. CNN-based and GCN-based) and two classifiers (i.e. Unet and P-GCN). Our AEI shows significant improvement when regarding human logical thinking to extract spatio-temporal visual representation. Our AEI robustly outperforms SOTA methods with remarkable performance and generalization for both temporal action proposal generation and temporal action detection.
App. Sci.

A Smart System for Text-Lifelog Generation from Wearable Cameras in Smart Environment Using Concept-Augmented Image Captioning with Modified Beam Search Strategy

Khoa Vo, Quoc-An Luong, Duy-Tam Nguyen, Mai-Khiem Tran, and Minh-Triet Tran

Applied Sciences Jan 2019

Abs HTML

During a lifetime, a person can have many wonderful and memorable moments that he/she wants to keep. With the development of technology, people now can store a massive amount of lifelog information via images, videos or texts. Inspired by this, we develop a system to automatically generate caption from lifelog pictures taken from wearable cameras. Following up on our previous method introduced at the SoICT 2018 conference, we propose two improvements in our captioning method. We trained and tested the model on the baseline MSCOCO datasets and evaluated on different metrics. The results show better performance compared to our previous model and to some other image captioning methods. Our system also shows effectiveness in retrieving relevant data from captions and achieve high rank in ImageCLEF 2018 retrieval challenge.
IEEE Access

ABN: Agent-Aware Boundary Networks for Temporal Action Proposal Generation

Khoa Vo, Kashu Yamazaki, Sang Truong, Minh-Triet Tran, Akihiro Sugimoto, and Ngan Le

IEEE Access Jan 2021

Abs HTML

Temporal action proposal generation (TAPG) aims to estimate temporal intervals of actions in untrimmed videos, which is a challenging yet plays an important role in many tasks of video analysis and understanding. Despite the great achievement in TAPG, most existing works ignore the human perception of interaction between agents and the surrounding environment by applying a deep learning model as a black-box to the untrimmed videos to extract video visual representation. Therefore, it is beneficial and potentially improves the performance of TAPG if we can capture these interactions between agents and the environment. In this paper, we propose a novel framework named Agent-Aware Boundary Network (ABN), which consists of two sub-networks: (1) an Agent-Aware Representation Network to obtain both agent-agent and agents-environment relationships in the video representation; and (2) a Boundary Generation Network to estimate the confidence score of temporal intervals. In the Agent-Aware Representation Network, the interactions between agents are expressed through local pathway, which operates at a local level to focus on the motions of agents whereas the overall perception of the surroundings are expressed through global pathway, which operates at a global level to perceive the effects of agents-environment. Comprehensive evaluations on 20-action THUMOS-14 and 200-action ActivityNet-1.3 datasets with different backbone networks (i.e C3D, SlowFast and Two-Stream) show that our proposed ABN robustly outperforms state-of-the-art methods regardless of the employed backbone network on TAPG. We further examine the proposal quality by leveraging proposals generated by our method onto temporal action detection (TAD) frameworks and evaluate their detection performances.