Advanced Neural Implants
and Control
mind control chips,
and subdermal interfaces already in use,
please view blue link above for images
Daryl R. Kipke
Associate Professor
Department of Bioengineering
Arizona State University
Tempe, AZ 85287
kipke@asu.edu
for full details and images see blue link above
Approved for Public Release, Distribution Unlimited: 01-S-1097
The Underlying Premise…
The ability to engineer reliable,
high-capacity direct interfaces to the
brain and then integrate these into a
host of new technologies will cause
the world of tomorrow to be much
different than that of today.
However…
� There are some serious scientific barriers between
where we stand today and where we can stand in
the future.
• How do we establish permanent and reliable interfaces to
selected areas of the central nervous system?
• How do we use these interfaces to directly and reliably
communicate at high rates with the brain?
Applied Neural Implants and Control
Project Director
Kipke (BME)
Advisory
Committee
Raupp, Hoppensteadt, Farin
Visualization &
Modeling
Farin (CSE)
Nelson (CSE)
Razdan (CSE)
Smith (Math)
Systems Science &
Signal Processing
He (BME)
Hoppensteadt (Math & EE)
Kipke (BME)
Si (EE)
Neural & Tissue
Engineering
Kipke (BME)
Massia (BME)
Panitch (BME)
Rousche (BME)
Tissue Culture &
Analysis
Capco (Bio)
Massia (BME)
Pauken (Bio)
Materials Synthesis
& Bioactive
Coatings
Ehestraimi (BME)
Massia (BME)
Panitch (BME)
Raupp (ChemE)
MEMS
Shen (EE)
Pivin (EE)
Li (EE)
INFO
BIO
MICRO
Primary Goals of the
BIO:INFO:MICRO Project
� Develop new neural implant
technologies to establish
reliable, high-capacity, and long-term
information channels
between the brain and external
world.
� Develop real-time signal
processors and system
controllers to optimize
information transmission
between the brain and the
external world.
SysSci VizMod
NeuEng MEMS
TisClt
Mat'lSyn
Systems-level Approach…
Feedback control signals
Subject
Neural system
(global)
Controlled
neural
plasticity
local
Neural
Implant
Adaptive
Controller
External
World
Objective 2: Optimize
Adaptive Controller
Objective 1: Optimize neural
interface
Topics
� Project overview
� Towards the Development of Next-generation
Neural Implants (BIO, MICRO,
and INFO)
� Bioactive Coatings to Control the Tissue
Responses to Implanted Microdevices
� Modeling the Device-Tissue Interface
� Direct Cortical Control of an Actuator
� Neural Control of Auditory Perception
� Wrap-up
Focus on Next-Generation
Neural Implants
Feedback signals: local
Subject
Neural system
(global)
Controlled
neural
plasticity
local
Neural
Implant
Neural
Controller
External
World
host response
Info. Signals: electrical
& chemical
Objective 2: Optimize
Adaptive Controller
Objective 1: Optimize neural
interface to achieve reliable, two-way,
high-capacity information channels.
…and “self-diagnostic”
Fundamental Problem of Implantable
Microelectrode Arrays
� Brain often encapsulates the device with scar tissue
� Normal brain movement may cause micro-motion at the tissueelectrode
interface
� Proteins adsorb onto device surface
� Useful neural recordings are eventually lost
Electrode 1
Electrode N
Implant Failure
Month 1 Implant
Month N
3rd-Generation Neural Implants
Technology
Spectrum
1st-generation
Microwires
2nd-generation
Silicon arrays
3rd-generation
Neural Implants
Desired Properties
• Very high channel count
(<1000)
• Bioactive coatings
• Flexible
• Engineered surfaces
• Controlled biological
response
• Integrated electronics
“Brain-centered” Design of Neural Implants
Initial conceptual designs
B
B
A
A
A A B B
through hole connecting
channel recording site
bioactive gel
Standard Perforated Probe
Simple Bioactive Probe
Differential Bioactive Probe
recording site through hole
bioactive gel
flexible
polyimide
substrate
bond pads
e.g. corticosteroid NGF e.g. GABA
cross-section (A-A) cross-section (B-B)
Polymer-substrate Neural Implants
• 2-D planar devices can be bent into 3-D structures
• Increases insertion complexity
Holes to
promote
integration
with neuropil 90 degree
angles
Recordings From Polymer-substrate
Neural Implants
Chan. 9
Chan. 10
One Day Post-op
Lost most unit activity
after 7 days – Most likely
due to failure to properly
close dural opening.
Flexible Neural Implants Present
Surgical Challenges
� While the “micro-motion” hypothesis suggests that flexible
neural implants should be more stable, the same flexibility
presents significant new surgical challenges.
“Difficult” insertion “Easy” insertion
Rdr2, 9-00 Rdr3, 9-00
Using Dissolvable Coatings to
Stiffen the Neural Implant
� Dip-coat microdevice with polyethylene glycol (PEG)
• Provides mechanical stiffening prior to implant
• Quickly dissolves when in contact with tissue
First insertion of coated microdevice into Second insertion of coated microdevice
gelatin -- Device easily penetrates into gelatin – The device is too flexible to
material penetrate material because the PEG has
dissolved.
Micromachined Surgical Devices
Vacuum nozzle
Flexible probe
Insertion aid
Vacuum Actuated Knife/Inserter
PEG
Silicon Knife/Inserter
Exploratory Functionality
Bioactive Component
Storage Structures
Passive
Surface Engineering
Active
FET Devices,
ChemFETs
Electrical
Recording/Stimulating
Surfaces
Other Active Devices
(Thermal, Magnetic, Strain, etc.)
Fluid Microchannels
Polymer Substrate
• Magnetic/thermal
stimulation
• Drug delivery channels
• Active micromanipulation
of probes
Currently...
Internal Review
Feasibility Studies
Insertion Aids
Mechanical
Transfer Structures
Signal
Processing Termination
Multiple Dimensions and Forms
Implant Coatings and Surface Modifications
Parylene-N,C Photo-crosslinked
Cl Polyimides
Cl O O
O
n
C
C
C
C
smooth porous
N N
O O
Surface Plasma Treatments NH2 NH2 NH2 NH2
(NH3 - Amination)
Advanced Neuro-Device Interfaces
Passive
Chemical/Electronic
NH2 NH2 NH2 NH2
Amplification
ion beam
metal
modified region site or interdigits
release layer polymer (PI/P-C)
or substrate
Active
Silicon FETs?
Topics
� Project overview
� Towards the Development of 3rd-Generation
Neural Implants (BIO, MICRO, and INFO)
� Bioactive Coatings for Controlled
Biological Response (BIO, MICRO, and INFO)
� Modeling the Device-Tissue Interface
� Direct Cortical Control of an Actuator
� Neural Control of Auditory Perception
� Wrap-up
Approach
Advanced biomaterials and
micro-devices for long-term
implants (BIO, MICRO, INFO)
Cellular and biochemical
response characterization
(BIO, MICRO)
Models and 3-D visualization
of device-tissue dynamics
(BIO, INFO)
Engineer the neural implant surface in order to control
both the material response and the host response.
Factors Limiting Chronic
Soft Tissue Implants
� Inability to control cellular interactions at
biomaterial-tissue interface
� Initial adsorption of biological proteins
• Non-selective cellular adhesion
� Unavoidable “generic” foreign body reactions
• Inflammation
• Fibrous capsule formation
Potential Solution
� Engineer surface for minimal protein adsorption
and selective cell adhesion
• Cell-resistant polymer coatings
• Synthetic: Polyethylene Glycol, Polyvinyl Alcohol
• Natural: Polysaccharides, Phospholipids
• Surface immobilization of biologically active
molecules
• Mimic biochemical signals of extracellular matrix
• Cell binding domains for integrin receptors
Biomimetic Surface Modification
NH2 NH2
OH
O
HO
N
O
O
OH
OH
O
O
OH
OH
O
HO HO HO
O
OH
OH
O O
HO
HO N OH
O
HO
O
NTF NTF
Material Surface
Recombinant NGF Fusion Protein
Factor IIIa
Active or inactive plasmindegradable
substrate
Degraded plasminsubstrate
substrate
Human b-NGF
plasmin
Fibrin
Plasmin
cleavage Human b-NGF
Fibrin
Bioactive Functionality
Methods 6-hour diffusion in rat cortex
Fluorescence Intensity Profile
250
NeuroTrace� DiI tissue-labeling paste,
inverted fluorescent microscope with
FITC/rhodamine filter cube
200
150
Pixel Value
100
5 0
0
0 20 40 60 80 100 120 140 160
Distance (microns)
Topics
� Project overview
� Towards the Development of 3rd-Generation
� Bioactive Coatings to Control the Tissue
Neural Implants (BIO, MICRO, and INFO)
Responses to Implanted Microdevices (BIO, MICRO,
and INFO)
� Modeling the Device-Tissue Interface
(BIO, MICRO, and INFO)
� Direct Cortical Control of a Motor Prosthesis
� Neural Control of Auditory Perception
� Wrap-up
The Device-Tissue Interface
Neural Interface:
Micro-device, Neurons, Glia, Extracellular Space
The Goal is to Characterize, Predict, and Control
the Device-Tissue Interface
Tissue State
(e.g., encapsulation, excitability)
Biophysical
Model of the
Device-Tissue
Interface
Device Function
(e.g., impedance spectrum)
• Integrate bioelectrical, histological and biochemical data
• Optimize electrode specifications
Visualization of the Chronic Device-Tissue
Interface With Confocal Microscopy
A B
C D
In vivo Visualization of the Chronic
Device-Tissue Interface
Multi-Domain Continuum Model
( )
( )
( )
/
/
/
At each "point" in space:
volume fraction
potential ,
conductivity tensor
membrane parameters
, , , etc.
ei
ei
ei
L
r
f
rt
G
a C g
F
r
r
r
r
• Tissue is two (or more) coupled
volume-conducting media
• Electrode is boundary condition
r
r
Equations for a Multi-Domain
Continuum Model
Volume conductor equations (conservation of current)
- fe��(Ge�Fe ) = +�Imemi
+ Iapp
i
- fi
��(Gi
�Fi ) = -Imemi
i = index over intracellular domains
Membrane potential(s) and membrane current(s)
� ¶V � Vi
= F i -F e
I
memi
= ai �
Ł Ci ¶t
i + Iioni �
ł
-1
F= potential (mV) ai = surface to volume ratio (cm) Vi = membrane potential (mV) ei /
3 2
Gei = conductivity (mS/cm) Imemi
= membrane current (mA/cm) Ci = membrane capacitance (mF/cm ) /
3 2 fei = volume fraction Iapp = applied current (m A/cm) Iioni = membrane current (mA/cm ) /
Levels of Modeling
Numerical
Multiple intracellular domains
Voltage-dependent conductances
ioni
= � g ij � qijk (Vi - E j )
j k
¥ ¶qijk qijk -q V ijk ( i )
= -
¶t tijk (Vi )
Complex electrode geometry
Tissue inhomogeneous and
anisotropic
under construction
Analytical
A single intracellular domain
Passive membrane conductance
Iion
= g L (V E - L )
Simple electrode geometry
Tissue assumed homogenous and
isotropic
much progress
I
Bi-domain Model for the
Microcapillary Bioreactor
Calculate profiles
F1
ei / (x;w)
in bioreactor
...and impedance...
Z(w) =
F1
e ( L;w) -F1
e (0;w)
j
1
100 Hz
1 Write BCs and assume: 1 / (, ) ( ; ) i t i t
e i ei j j e x t x w
= � F = F w
Z
w
( )
/
...and predict
as tissue parameters
, , , , ,
are experimentally
manipulated
e i e i L
Z
f G C g E
w
a
V
F e
F i E L
/ ew
/ L
Recap
� Focused & integrated effort
• BioMEMS…Neural
Engineering…Materials…
Computational
Neuroscience…Cellular
Biology…Visualization
� Why are we so excited?
• We have the very real
potential of characterizing
the biological responses to
neural implants and then
engineering new classes
of microdevices to provide
a permanent high-capacity
interface to the brain
BIO
INFO
MICRO
Why the BIO, INFO, and
MICRO Program?
� Wide-open Challenges
• Characterizing and modeling the biological (cellular and
chemical) responses around a neural implant
• Controlling the dynamic biological responses around a neural
implant.
• Designing, fabricating, and using “advanced” neural implants
� Collaboration Possibilities
• Additional functionalities for implantable microdevices of the
class that we are working on.
• Exploring fundamentally new types of tissue-device interfaces.
• Complementary studies of the neural interface (experimental
and analytical)
• Confocal microscopy of the neural interface
• Sharing technologies, procedures, insights, etc…
• New emergent ideas…
Systems-level Analysis of Advanced
Neuroprosthetic Systems
Feedback control signals
Subject
Neural system
(global)
Controlled
neural
plasticity
local
Neural
Implant
Adaptive
Controller
External
World
Objective 2: Optimize
Adaptive Controller
Objective 1: Optimize neural
interface
Systems-level Approach for Advanced
Neuroprosthetic Systems
Subject
Neural system
(global)
Controlled
neural
plasticity
local
Neural
Implant
Adaptive
Controller
External
World
Feedback control signals
Objective 2: Develop Objective 1: Optimize neural
adaptive controller to interface
optimize system
performance.
Advanced Neuroprosthetic Systems
High-Level
Neural
Computation
Sensory
Transduction &
Pre-processing Motor
Commands
Movement
Perception,
Decision,
Detection
Sensory
Integration
External World
Neuroprosthetic System
� Underlying System Principles
•Two-way communication with targeted neural systems
•Harness neural plasticity to our advantage
•Appropriately balanced “wet-side” and “dry-side” computation
Approach
� Four Project Areas
�Direct neural control of actuators
�Detection of novel sensory stimuli through
monitoring neural activity
�Neural control of behavior
�Investigate signal transformations from
ensembles of single neurons to local field
potentials to EEG.
Topics
�
�
�
�
Project overview
Towards the Development of 3rd-Generation Neural
Implants (BIO, MICRO, and INFO)
Bioactive Coatings to Control the Tissue Responses to
Implanted Microdevices (BIO, MICRO, and INFO)
Modeling the Device-Tissue Interface (BIO, MICRO, and
INFO)
� Direct Cortical Control of a Motor Prosthesis
(BIO, MICRO, and INFO)
� Neural Control of Auditory Perception
� Wrap-up
Direct Cortical Control of Actuators
High-Level
Neural
Computation
Sensory
Transduction &
Pre-processing Motor
Commands
Movement
Perception,
Decision,
Detection
Sensory
Integration
External World
Neuroprosthetic
System
Goal: Control
arm-related
actuator
External Actuator
Robotic Arm or
Virtual Reality
Fundamental Questions
� What are “optimal” real-time signal processing
strategies for precise 3-D control of external, armrelated
actuators in the presence of sensory
distractions and/or physical perturbations to the
arm?
� To what extent can we use composite neural
signals [neuronal (unit) recordings, local field
potentials, and brain-surface recordings] for control
signals?
� How do we take advantage of inherent or
controlled neural plasticity in order to optimize
system performance?
Experimental Preparation
• Train monkeys to perform tracking and/or reaching tasks.
• Record cortical responses with multichannel neural
implants.
• Measure arm movement in 3-D space.
Chronic Neural Recordings
� Multi-channel neural implants in motor and sensorimotor cortical areas.
� Eventually: Sub-dural electrodes for local potentials
-0.2 0 0.2 0.4 0.6
0
10
dsp009b
-0.2 0 0.2 0.4 0.6
0
20
40
dsp012a
-0.2 0 0.2 0.4 0.6
0
10
20 dsp018a
-0.2 0 0.2 0.4 0.6
0
20
40
dsp024a
-0.2 0 0.2 0.4 0.6
0
20
40
dsp025a
-0.2 0 0.2 0.4 0.6
Time (sec)
0
10
dsp030a
-0.2 0 0.2 0.4 0.6
0
100
dsp034a
-0.2 0 0.2 0.4 0.6
0
50
100
150
dsp037a
-0.2 0 0.2 0.4 0.6
0
5
10
15 dsp040a
-0.2 0 0.2 0.4 0.6
0
10
20
dsp042a
-0.2 0 0.2 0.4 0.6
0
20
dsp042b
-0.2 0 0.2 0.4 0.6
Time (sec)
0
10
20
30 dsp045a
-0.2 0 0.2 0.4 0.6
0
20
40
dsp046a
-0.2 0 0.2 0.4 0.6
0
5
10
15
dsp051a
-0.2 0 0.2 0.4 0.6
0
20
40
dsp057a
-0.2 0 0.2 0.4 0.6
0
40
80
dsp058a
Perievent Histograms Target 1, reference = C_rel, bin = 20 ms
Neural
Recording
System
Offline
Analysis
Real-time
Signal
Processing
Actuator
Control
Extracellular recordings
Direct Cortical Control of Movement
Green ball: Target Yellow ball: Actual hand position, or
hand position estimated from cortical
responses
m0602pa
Topics
�
�
�
�
�
Project overview
Towards the Development of 3rd-Generation Neural
Implants (BIO, MICRO, and INFO)
Bioactive Coatings to Control the Tissue Responses to
Implanted Microdevices (BIO, MICRO, and INFO)
Modeling the Device-Tissue Interface (BIO, MICRO, and
INFO)
Direct Cortical Control of a Motor Prosthesis (BIO, MICRO,
and INFO)
� Neural Control of Auditory Perception(BIO,
MICRO, and INFO)
� Wrap-up
Neural Control of Auditory Perception
High-Level
Neural
Computation
Sensory
Transduction &
Pre-processing Motor
Commands
Movement
Perception,
Decision,
Detection
Sensory
Integration
External World
Neuroprosthetic
System
Goal: Control
auditory perception
Fundamental Questions
� To what extent can we control auditory-mediated behavior using
intra-cortical microstimulation (ICMS) through the neural interface?
Transmitter Channel Receiver
Source
Signal
Received
Signal
Stimulator Neural
Interface
Auditory
Cortex
� What are the information transmission characteristics of the
multichannel neural implant in high-level cortical areas using
ICMS?
� Channel capacity (bits per second)
� Channel reliability
� Channel resolution
� How can we optimize information transmission
� Implant designs, Neural implant locations, Signal encoding strategies,
Controlled neural plasticity
Chronic Neural Recordings
� Multi-channel neural implants in primary auditory cortex
Extracellular recordings
in auditory cortex
Estimation of
Neural
Recording
System
Offline
Analysis
Neuronal
Response
Properties
Algorithm Selection
Signal
Encoder
Sounds Electrical
Stimulation
to Aud. Ctx.
Behavioral performance to both sounds and
cortical electrical stimulation
Auditory Behavior
• Lever-press sound or ICMS discrimination task
• Center paddle hit starts trial, 2-tone pair presented
• Reward obtained by signaling the correct stimulus
sequence
center left right
rat
Frequency
response areas
Frequency Selectivity in Auditory Cortex
1 2 5 10 20 30
40
60
80
dsp024b
28.
56.
1 2 5 10 20 30
40
60
80
dsp018d
11.
22.
1 2 5 10 20 30
40
60
80
dsp020a
10.
20.
1 2 5 10 20 30
40
60
80
dsp024a
10.
20.
1 2 5 10 20 30
40
60
80
dsp012a
21.
42.
1 2 5 10 20 30
40
60
80
dsp018b
10.5
21.
1 2 5 10 20 30
40
60
80
dsp018c
22.
44.
1 2 5 10 20 30
40
60
80
dsp002a
3.
6.
1 2 5 10 20 30
40
60
80
dsp002b
5.5
11.
1 2 5 10 20 30
40
60
80
dsp010b
12.
24.
Freq.
Sound
Level
Signal Encoding Algorithm:
Frequency Selectivity
ICMS pattern is based
solely on frequency
selectivity of neurons
recorded on an electrode
dB
80
60
40
u5b 8
6
Spikes
4
2
0 1 5 10 30
kHz
u32a
0
2
4
6
8
kHz
1 5 10 30
Spikes
80
60 dB
40
Behavioral Performance
Ricms6
Rat Behavioral Performance
RICMS 6
100
09/06/00
09/16/00
09/26/00
10/06/00
10/16/00
10/26/00
Training day
Implanted
90
80
Percent Correct
70
60
50
40
30
20
10
0
Cortical
Electrodes
D
Expected Results to ICMS Stimuli
Begin ICMS
100
% D% due
to ICMS
Trial #
Auditory trial =
ICMS Algorithm1 =
ICMS Algorithm2 =
Behavioral Curve
RICMS 6 10/25 (Only Session)
100
80
Percentage
audPercent,
icmsPercent,
60
40
20
0
0 100 200
Trial
Alternative Signal Encoding Algorithm:
Cortical Activation Pattern
For a given electrode, the unit firing pattern is used as a
template for ICMS delivery
Auditory
Stimulus Sound on
Response
Raster
Matching ICMS
‘pattern’
***Procedure is simultaneously duplicated on each active electrode
Recap
� Focused & integrated effort
• Neural Engineering…Signal
Processing…Systems
Neurophysiology…Visualization
� Why are we so excited?
• We have the very real
potential of developing new
classes of neuroprosthetic
systems to explore our ability
to interact directly with the
brain.
BIO
INFO
MICRO
BIO, INFO, and MICRO…
� Wide-open Challenges
• Appropriate mathematical constructs for describing neural
encoding and decoding.
• Advanced data visualization techniques for understanding this
new class of neural data.
• Understanding signal transformations as a function of the
spatial and temporal scale of the neural data.
� Collaboration Possibilities
• Exploring new signal encoding and decoding strategies for
particular neuroprosthetic applications.
• Sharing technologies, procedures, insights, etc…
• New emergent ideas…
Topics
�
�
�
�
�
�
Project overview
Towards the Development of 3rd-Generation Neural
Implants (BIO, MICRO, and INFO)
Bioactive Coatings to Control the Tissue Responses to
Implanted Microdevices (BIO, MICRO, and INFO)
Modeling the Device-Tissue Interface (BIO, MICRO, and
INFO)
Direct Cortical Control of a Motor Prosthesis (BIO, MICRO,
and INFO)
Neural Control of Auditory Perception(BIO, MICRO, and
INFO)
� Wrap-up
Project Challenges
� Scientific
• Overcoming engineering and scientific hurdles.
• Identifying and fostering strategic alliances with appropriate
external groups.
• Crossing disciplines
� Management
• Strategic planning
• Resource allocation
• Open and effective communication among the diverse project
team
• Team-building: Maintaining enthusiasm, energy, and focus
after the initial “honeymoon” period
“Insanely Intense
Interdisciplinary” Research
“pieces of a puzzle” “easy synergism”
BIO
INFO
MICRO
MICRO BIO
INFO
Breakthrough
Science
•Hard work
•Open minds
•Honesty
•Top-notch research
--
What Does the Future Hold?
“Perhaps within 25 years there will be some new ways to put
information directly into our brains. With the implant technology that
will be available by about 2025, doctors will be able to put something
like a chip in your brain to prevent a stroke, stop a blood clot, detect
an aneurysm, help your memory or treat a mental condition. You
may be able to stream (digital) information through your eyes to the
brain. New drugs may enhance your memory and fire up your
neurons.” Dr. Arthur Caplan,
Director of the Center of Bioethics,
University of Pennsylvania
Arizona Republic, Dec 27, 1998.
Acknowledgments
� ASU Colleagues
• 13 co-PI’s, 5 research faculty, numerous graduate
and undergraduate students.
� Arizona State University administration
• Seed funding from Department, College, and
University
• Significant cost-share on this project
� DARPA Program Managers
• Eric Eisenstadt, Abe Lee, and Gary Strong