LeaFleT: LEArning neural networks with FLExible nonlinearities by Tensor methods

In November 2024, we organized a two-day workshop on LOw-Rank Approximations and their Interactions with Neural NEtworks, a closing event for the LeaFleT project

Context and summary

Neural networks are a fundamental tool for solving various artificial intelligence tasks, such as supervised and unsupervised classification. Recent progress is linked to deep neural networks with extremely large number of layers, which helped to achieve remarkable results in the context of many applied tasks for analysis and interpretation of natural images, audio signals and textual data. Despite this success, they still have a number of drawbacks, including lack of interpretability and large number of parameters. In this project, we propose to simplify neural network architectures by allowing flexible nonlinear activation functions, contrary to fixed activation functions typically used. The proposed pathway is based on an original tensor-based technique for decomposition of multivariate maps, developed in the context of nonlinear system identification. The fundamental property of such decompositions is identifiability, which we hope would be transferrable in the deep learning setup. We believe that the identifiability property, by enforcing stability of the representation, could be one of possible ways to define interpretability of neural networks. Also, neural networks will be simplified by reducing the number of layers, and recent advances in tensor computations can be used. The main goal of this project is to develop effective learning tools for the proposed model, together with their publicly available software implementation.

Collaborators

Postdocs

Publications of LeaFleT

Conference papers

Low-rank updates of pre-trained weights for multi-task learning
Audibert, A., Amini, M., Usevich, K. and Clausel, M.
In Findings of the Association for Computational Linguistics, ACL 2023, July, 2023, pp. 7544-7554, Association for Computational Linguistics
[BibTeX] [DOI] [preprint]

@inproceedings{audibert:hal-04360908,
  author = {Audibert, Alexandre and Amini, Massih and Usevich, Konstantin and Clausel, Marianne},
  title = {Low-rank updates of pre-trained weights for multi-task learning},
  booktitle = {Findings of the Association for Computational Linguistics, ACL 2023},
  publisher = {Association for Computational Linguistics},
  year = {2023},
  pages = {7544-7554},
  url = {https://hal.science/hal-04360908},
  doi = {https://doi.org/10.18653/v1/2023.findings-acl.476}
}

Compressing Neural Networks with Two-Layer Decoupling
De Jonghe, J., Usevich, K., Dreesen, P. and Ishteva, M.
In 2023 IEEE 9th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), December, 2023, pp. 226-230
[BibTeX] [DOI] [preprint]

@inproceedings{usevich:hal-05008394,
  author = {Joppe De Jonghe and  Konstantin Usevich and Philippe Dreesen and Mariya Ishteva},
  title = {Compressing Neural Networks with Two-Layer Decoupling},
  booktitle = {2023 IEEE 9th International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP)},
  year = {2023},
  pages = {226-230},
  url = {https://hal.science/hal-05008394},
  doi = {https://doi.org/10.1109/CAMSAP58249.2023.10403509}
}

Tensor-based two-layer decoupling of multivariate polynomial maps
Usevich, K., Zniyed, Y., Ishteva, M., Dreesen, P. and de Almeida, A.
In 31st European Signal Processing Conference, EUSIPCO 2023, September, 2023, European Association for Signal Processing
[BibTeX] [DOI] [preprint]

@inproceedings{usevich:hal-04360913,
  author = {Usevich, Konstantin and Zniyed, Yassine and Ishteva, Mariya and Dreesen, Philippe and de Almeida, André},
  title = {Tensor-based two-layer decoupling of multivariate polynomial maps},
  booktitle = {31st European Signal Processing Conference, EUSIPCO 2023},
  publisher = {European Association for Signal Processing},
  year = {2023},
  url = {https://hal.science/hal-04360913},
  doi = {https://doi.org/10.23919/EUSIPCO58844.2023.10289900}
}

Low-rank tensor recovery for Jacobian-based Volterra identification of parallel Wiener-Hammerstein systems
Usevich, K., Dreesen, P. and Ishteva, M.
In 19th IFAC Symposium on System Identification, SYSID 2021, July, 2021
[BibTeX] [preprint]

@inproceedings{usevich:hal-03349340,
  author = {Usevich, Konstantin and Dreesen, Philippe and Ishteva, Mariya},
  title = {Low-rank tensor recovery for Jacobian-based Volterra identification of parallel Wiener-Hammerstein systems},
  booktitle = {19th IFAC Symposium on System Identification, SYSID 2021},
  year = {2021},
  url = {https://hal.science/hal-03349340}
}

Learning nonlinearities in the decoupling problem with structured CPD
Zniyed, Y., Usevich, K., Miron, S. and Brie, D.
In 19th IFAC Symposium on System Identification, SYSID 2021, July, 2021
[BibTeX] [preprint]

@inproceedings{zniyed:hal-03223831,
  author = {Zniyed, Yassine and Usevich, Konstantin and Miron, Sebastian and Brie, David},
  title = {Learning nonlinearities in the decoupling problem with structured CPD},
  booktitle = {19th IFAC Symposium on System Identification, SYSID 2021},
  year = {2021},
  url = {https://hal.science/hal-03223831}
}

Tensor-based approach for training flexible neural networks
Zniyed, Y., Usevich, K., Miron, S. and Brie, D.
In 55th Asilomar Conference on Signals, Systems and Computers, October, 2021
[BibTeX] [preprint]

@inproceedings{zniyed:hal-03518648,
  author = {Zniyed, Yassine and Usevich, Konstantin and Miron, Sebastian and Brie, David},
  title = {Tensor-based approach for training flexible neural networks},
  booktitle = {55th Asilomar Conference on Signals, Systems and Computers},
  year = {2021},
  url = {https://hal.science/hal-03518648}
}

On the convergence of Jacobi-type algorithms for Independent Component Analysis
Li, J., Usevich, K. and Comon, P.
In SAM 2020 - 11th Sensor Array and Multichannel Signal Processing Workshop, June, 2020
[BibTeX] [DOI] [preprint]

@inproceedings{li:hal-02994725,
  author = {Li, Jianze and Usevich, Konstantin and Comon, Pierre},
  title = {On the convergence of Jacobi-type algorithms for Independent Component Analysis},
  booktitle = {SAM 2020 - 11th Sensor Array and Multichannel Signal Processing Workshop},
  year = {2020},
  url = {https://hal.science/hal-02994725},
  doi = {https://doi.org/10.1109/SAM48682.2020.9104331}
}

On Cramér-Rao lower bounds with random equality constraints
Prévost, C., Chaumette, E., Usevich, K., Brie, D. and Comon, P.
In ICASSP proceedings, May, 2020
[BibTeX] [DOI] [preprint]

@inproceedings{prevost:hal-02486600,
  author = {Prévost, Clémence and Chaumette, Eric and Usevich, Konstantin and Brie, David and Comon, Pierre},
  title = {On Cramér-Rao lower bounds with random equality constraints},
  booktitle = {ICASSP proceedings},
  year = {2020},
  url = {https://hal.science/hal-02486600},
  doi = {https://doi.org/10.1109/ICASSP40776.2020.9054031}
}

Journal papers

Personalized coupled tensor decomposition for multimodal data fusion: Uniqueness and algorithms
Borsoi, R.A., Usevich, K., Brie, D. and Adali, T.
IEEE Transactions on Signal Processing, 2025, Vol. 73, pp. 113 - 129, Institute of Electrical and Electronics Engineers
[BibTeX] [DOI] [preprint]

@article{borsoi:hal-04822106,
  author = {Borsoi, Ricardo Augusto and Usevich, Konstantin and Brie, David and Adali, Tülay},
  title = {Personalized coupled tensor decomposition for multimodal data fusion: Uniqueness and algorithms},
  journal = {IEEE Transactions on Signal Processing},
  publisher = {Institute of Electrical and Electronics Engineers},
  year = {2025},
  volume = {73},
  pages = {113 - 129},
  url = {https://hal.science/hal-04822106},
  doi = {https://doi.org/10.1109/TSP.2024.3510680}
}

Gaussian process regression in the flat limit
Barthelme, S., Amblard, P.-O., Tremblay, N. and Usevich, K.
Annals of Statistics, December, 2023, Vol. 51 (6), pp. 2471-2505, Institute of Mathematical Statistics
[BibTeX] [DOI] [preprint]

@article{barthelme:hal-03869191,
  author = {Barthelme, Simon and Amblard, Pierre-Olivier and Tremblay, Nicolas and Usevich, Konstantin},
  title = {Gaussian process regression in the flat limit},
  journal = {Annals of Statistics},
  publisher = {Institute of Mathematical Statistics},
  year = {2023},
  volume = {51},
  number = {6},
  pages = {2471-2505},
  url = {https://hal.science/hal-03869191},
  doi = {https://doi.org/10.1214/23-AOS2336}
}

Determinantal point processes in the flat limit
Barthelme, S., Tremblay, N., Usevich, K. and Amblard, P.-O.
Bernoulli, May, 2023, Vol. 29 (2), pp. 957-983, Bernoulli Society for Mathematical Statistics and Probability
[BibTeX] [DOI] [preprint]

@article{barthelme:hal-03359889,
  author = {Barthelme, Simon and Tremblay, Nicolas and Usevich, Konstantin and Amblard, Pierre-Olivier},
  title = {Determinantal point processes in the flat limit},
  journal = {Bernoulli},
  publisher = {Bernoulli Society for Mathematical Statistics and Probability},
  year = {2023},
  volume = {29},
  number = {2},
  pages = {957-983},
  note = {Most of this material first appeared in arXiv:2007.04117, which has been split into two. The presentation has been simplified and some material is new},
  url = {https://hal.science/hal-03359889},
  doi = {https://doi.org/10.48550/arXiv.2107.07213}
}

Convergence of gradient-based block coordinate descent algorithms for non-orthogonal joint approximate diagonalization of matrices
Li, J., Usevich, K. and Comon, P.
SIAM Journal on Matrix Analysis and Applications, April, 2023, Vol. 44 (2), pp. 592-621, Society for Industrial and Applied Mathematics
[BibTeX] [DOI] [preprint]

@article{li:hal-03408912,
  author = {Li, Jianze and Usevich, Konstantin and Comon, Pierre},
  title = {Convergence of gradient-based block coordinate descent algorithms for non-orthogonal joint approximate diagonalization of matrices},
  journal = {SIAM Journal on Matrix Analysis and Applications},
  publisher = {Society for Industrial and Applied Mathematics},
  year = {2023},
  volume = {44},
  number = {2},
  pages = {592-621},
  url = {https://hal.science/hal-03408912},
  doi = {https://doi.org/10.1137/21M1456972}
}

Extended L-ensembles: A new representation for determinantal point processes
Tremblay, N., Barthelme, S., Usevich, K. and Amblard, P.-O.
The Annals of Applied Probability, February, 2023, Vol. 33 (1), pp. 613-640, Institute of Mathematical Statistics (IMS)
[BibTeX] [DOI] [preprint]

@article{tremblay:hal-03359895,
  author = {Tremblay, Nicolas and Barthelme, Simon and Usevich, Konstantin and Amblard, Pierre-Olivier},
  title = {Extended L-ensembles: A new representation for determinantal point processes},
  journal = {The Annals of Applied Probability},
  publisher = {Institute of Mathematical Statistics (IMS)},
  year = {2023},
  volume = {33},
  number = {1},
  pages = {613-640},
  url = {https://hal.science/hal-03359895},
  doi = {https://doi.org/10.1214/22-AAP1824}
}

Robust eigenvectors of symmetric tensors
Muller, T., Robeva, E. and Usevich, K.
SIAM Journal on Matrix Analysis and Applications, December, 2022, Vol. 43 (4), pp. 1784-1805, Society for Industrial and Applied Mathematics
[BibTeX] [DOI] [preprint]

@article{muller:hal-04312602,
  author = {Muller, Tommi and Robeva, Elina and Usevich, Konstantin},
  title = {Robust eigenvectors of symmetric tensors},
  journal = {SIAM Journal on Matrix Analysis and Applications},
  publisher = {Society for Industrial and Applied Mathematics},
  year = {2022},
  volume = {43},
  number = {4},
  pages = {1784-1805},
  url = {https://hal.science/hal-04312602},
  doi = {https://doi.org/10.1137/21M1462052}
}

Hyperspectral super-resolution accounting for spectral variability: coupled tensor LL1-based recovery and blind unmixing of the unknown super-resolution image
Prévost, C., Borsoi, R.A., Usevich, K., Brie, D., Bermudez, J.C.M. and Richard, C.
SIAM Journal on Imaging Sciences, January, 2022, Vol. 15 (1), pp. 110-138, Society for Industrial and Applied Mathematics
[BibTeX] [DOI] [preprint]

@article{prevost:hal-03158076,
  author = {Prévost, Clémence and Borsoi, Ricardo A and Usevich, Konstantin and Brie, David and Bermudez, José C. M. and Richard, Cédric},
  title = {Hyperspectral super-resolution accounting for spectral variability: coupled tensor LL1-based recovery and blind unmixing of the unknown super-resolution image},
  journal = {SIAM Journal on Imaging Sciences},
  publisher = {Society for Industrial and Applied Mathematics},
  year = {2022},
  volume = {15},
  number = {1},
  pages = {110-138},
  url = {https://hal.science/hal-03158076},
  doi = {https://doi.org/10.1137/21M1409354}
}

Constrained Cramér–Rao bounds for reconstruction problems formulated as coupled canonical polyadic decompositions
Prévost, C., Usevich, K., Haardt, M., Comon, P. and Brie, D.
Signal Processing, September, 2022, Vol. 198, pp. 108573, Elsevier
[BibTeX] [DOI] [preprint]

@article{prevost:hal-03651874,
  author = {Prévost, Clémence and Usevich, Konstantin and Haardt, Martin and Comon, Pierre and Brie, David},
  title = {Constrained Cramér--Rao bounds for reconstruction problems formulated as coupled canonical polyadic decompositions},
  journal = {Signal Processing},
  publisher = {Elsevier},
  year = {2022},
  volume = {198},
  pages = {108573},
  url = {https://hal.science/hal-03651874},
  doi = {https://doi.org/10.1016/j.sigpro.2022.108573}
}

Spectral properties of kernel matrices in the flat limit
Barthelme, S. and Usevich, K.
SIAM Journal on Matrix Analysis and Applications, January, 2021, Vol. 42 (1), pp. 17-57, Society for Industrial and Applied Mathematics
[BibTeX] [DOI] [preprint]

@article{barthelme:hal-03017947,
  author = {Barthelme, Simon and Usevich, Konstantin},
  title = {Spectral properties of kernel matrices in the flat limit},
  journal = {SIAM Journal on Matrix Analysis and Applications},
  publisher = {Society for Industrial and Applied Mathematics},
  year = {2021},
  volume = {42},
  number = {1},
  pages = {17-57},
  note = {41 pages, 8 figures},
  url = {https://hal.science/hal-03017947},
  doi = {https://doi.org/10.1137/19M129677X}
}

Coupled tensor decomposition for hyper spectral and multispectral image fusion with inter-image variability
Borsoi, R.A., Prévost, C., Usevich, K., Brie, D., Bermudez, J.C.M. and Richard, C.
IEEE Journal of Selected Topics in Signal Processing, April, 2021, Vol. 15 (3), pp. 702-717, IEEE
[BibTeX] [DOI] [preprint]

@article{borsoi:hal-03106874,
  author = {Borsoi, Ricardo A and Prévost, Clémence and Usevich, Konstantin and Brie, David and Bermudez, José C M and Richard, Cédric},
  title = {Coupled tensor decomposition for hyper spectral and multispectral image fusion with inter-image variability},
  journal = {IEEE Journal of Selected Topics in Signal Processing},
  publisher = {IEEE},
  year = {2021},
  volume = {15},
  number = {3},
  pages = {702-717},
  url = {https://hal.science/hal-03106874},
  doi = {https://doi.org/10.1109/JSTSP.2021.3054338}
}

Jacobi-type algorithm for low rank orthogonal approximation of symmetric tensors and its convergence analysis
Li, J., Usevich, K. and Comon, P.
Pacific journal of optimization, September, 2021, Vol. 17 (3), pp. 357-379, Yokohama Publishers
[BibTeX] [DOI] [preprint]

@article{li:hal-03233467,
  author = {Li, Jianze and Usevich, Konstantin and Comon, Pierre},
  title = {Jacobi-type algorithm for low rank orthogonal approximation of symmetric tensors and its convergence analysis},
  journal = {Pacific journal of optimization},
  publisher = {Yokohama Publishers},
  year = {2021},
  volume = {17},
  number = {3},
  pages = {357-379},
  url = {https://hal.science/hal-03233467},
  doi = {https://doi.org/10.48550/arXiv.1911.00659}
}

Hyperspectral super-resolution with coupled Tucker approximation: Recoverability and SVD-based algorithms
Prévost, C., Usevich, K., Comon, P. and Brie, D.
IEEE Transactions on Signal Processing, January, 2020, Vol. 68, pp. 931-946, Institute of Electrical and Electronics Engineers
[BibTeX] [DOI] [preprint]

@article{prevost:hal-01911969,
  author = {Prévost, Clémence and Usevich, Konstantin and Comon, Pierre and Brie, David},
  title = {Hyperspectral super-resolution with coupled Tucker approximation: Recoverability and SVD-based algorithms},
  journal = {IEEE Transactions on Signal Processing},
  publisher = {Institute of Electrical and Electronics Engineers},
  year = {2020},
  volume = {68},
  pages = {931-946},
  url = {https://hal.science/hal-01911969},
  doi = {https://doi.org/10.1109/TSP.2020.2965305}
}

Approximate matrix and tensor diagonalization by unitary transformations: convergence of Jacobi-type algorithms
Usevich, K., Li, J. and Comon, P.
SIAM Journal on Optimization, October, 2020, Vol. 30 (4), pp. 2998-3028, Society for Industrial and Applied Mathematics
[BibTeX] [DOI] [preprint]

@article{usevich:hal-01998900,
  author = {Usevich, Konstantin and Li, Jianze and Comon, Pierre},
  title = {Approximate matrix and tensor diagonalization by unitary transformations: convergence of Jacobi-type algorithms},
  journal = {SIAM Journal on Optimization},
  publisher = {Society for Industrial and Applied Mathematics},
  year = {2020},
  volume = {30},
  number = {4},
  pages = {2998-3028},
  url = {https://hal.science/hal-01998900},
  doi = {https://doi.org/10.1137/19M125950X}
}

Preprints

A lifting approach to ParaTuck-2 tensor decompositions
Usevich, K.
, February, 2024
[BibTeX] [preprint]

@unpublished{usevich:hal-03966869,
  author = {Usevich, Konstantin},
  title = {A lifting approach to ParaTuck-2 tensor decompositions},
  year = {2024},
  note = {working paper or preprint},
  url = {https://hal.science/hal-03966869}
}

Computing asymptotic eigenvectors and eigenvalues of perturbed symmetric matrices
Usevich, K. and Barthelme, S.
, July, 2024
[BibTeX] [preprint]

@unpublished{usevich:hal-04659372,
  author = {Usevich, Konstantin and Barthelme, Simon},
  title = {Computing asymptotic eigenvectors and eigenvalues of perturbed symmetric matrices},
  year = {2024},
  note = {working paper or preprint},
  url = {https://hal.science/hal-04659372}
}

Two-layer decoupling of multivariate polynomials with coupled ParaTuck and CP decompositions
Usevich, K., Zniyed, Y., Ishteva, M., Dreesen, P. and de Almeida, A.L.F.
, February, 2023
[BibTeX] [preprint]

@unpublished{usevich:hal-03968630,
  author = {Usevich, Konstantin and Zniyed, Yassine and Ishteva, Mariya and Dreesen, Philippe and de Almeida, André L F},
  title = {Two-layer decoupling of multivariate polynomials with coupled ParaTuck and CP decompositions},
  year = {2023},
  note = {working paper or preprint},
  url = {https://hal.science/hal-03968630}
}

Tensor-based framework for training flexible neural networks
Zniyed, Y., Usevich, K., Miron, S. and Brie, D.
, June, 2021
[BibTeX] [preprint]

@unpublished{zniyed:hal-03273321,
  author = {Zniyed, Yassine and Usevich, Konstantin and Miron, Sebastian and Brie, David},
  title = {Tensor-based framework for training flexible neural networks},
  year = {2021},
  note = {working paper or preprint},
  url = {https://hal.science/hal-03273321}
}

Konstantin Usevich