A03 – Sequential and adaptive learning under dependence and non-standard objective functions

We consider the problem of learning sequentially, adaptively and on the basis of partial information
on an uncertain environment. In this setting, the learner sequentially and actively collects the data,
which is are not available before-hand in a batch form. The process is as follows : at each time t,
the learner chooses an action and receives a data point, that depends on the performed action. A
popular application—that we extensively considered in the two first periods—is the one of online
recommender systems. The objective of the recommender system (learner) is to recommend at
each time t an item (actions) to a user that fits their needs. The recommendation system does not
observe the preferences of the users, but only their opinions on what it (the recommendation
system) has recommended as items previously. This casts many interesting problems finding
optimal sequential strategies for focusing first on the most liked items, or spanning as well
as possible the preferences of the users, etc. The coupling of data assimilation and sequential
learning has a huge potential as it allows to make more efficient use of huge datasets coming
from dynamical systems.
In this field, the main aim is to design strategies for sequential data sampling that effectively
balance exploration and exploitation, considering dependencies between the data and a pre-
specified loss function. The main challenge is to gather information on the unknown environment
while minimising the loss by selecting actions that result in low expected loss. Key questions
include deriving bounds for the expected loss.
The main challenge that we considered in the two first periods was to deal with the fact that in
this application—and in many other real life examples—the data that the learner collects present
dependencies. In the first period, we focused on discrete switching systems. In the second period,
we focused on more complex systems, be it in regard to the nature of the system itself—namely
with continuous actions, or with a complex structure—or of the changes—no well-defined change
point, and more flexibility in the nature of the change.
In the last funding period, we want to investigate three complex problems where the outputs
of the first and second funding period will be instrumental. These problems involve optimising
the use of side-information between arms, combining topological properties, and dealing with
dependencies. The focus will be on: (1) exploring a graph with side-information on neighbouring
nodes, (2) learning a continuous, complex function at multiple points simultaneously under shape
constraints and dependency and (3) to employ sequential learning techniques for learning an
optimal randomised observation strategy compatible with classical filtering frameworks.
The first two problems necessitate, in two different ways, to make optimal usage of side-
information between the actions. This side-information arises from a mixture between topological
properties of the underlying sequential learning problem. This builds on the ideas developed
during the second funding period, where the continuous bandit setting was explored, as well as
on the foundational work on dependence in bandit settings, which was extensively studied during
the first and second funding periods. We want to optimise the usage of this side-information in
more complex problems than what we considered before, namely:
1 exploring a graph with side-information on neighbouring nodes, and potentially dependence
and missing information over the graph structure,
2 Learning a continuous function of low intrinsic dimension—or in the multi-view model—under
dependence assumption,
Furthermore, we aim to build on a pilot study [1] that explores the use of sequential learning
strategies to improve the subspace of the state space most relevant for the Bayesian inference step.
To this end, we will:
3 investigate randomised observation strategies for filtering.
For all three challenges, side information and dependencies will together form a complex
framework, requiring optimal strategies that take both elements and their interplay into account.